Oct 30 14:44:50.183502 ip-10-0-138-143 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 14:44:50.183514 ip-10-0-138-143 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 14:44:50.183522 ip-10-0-138-143 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 14:44:50.183836 ip-10-0-138-143 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 14:45:00.306303 ip-10-0-138-143 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 14:45:00.306320 ip-10-0-138-143 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8be41e4917e94e9a8f147bcc2aa83678 --
Oct 30 14:46:48.224279 ip-10-0-138-143 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 14:46:48.652008 ip-10-0-138-143 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:46:48.652008 ip-10-0-138-143 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 14:46:48.652008 ip-10-0-138-143 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:46:48.652008 ip-10-0-138-143 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657738    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657845    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657851    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657856    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657859    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.657850 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657864    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657868    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657871    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657875    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657879    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657885    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657889    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657893    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657901    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657904    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657908    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657912    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657916    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657924    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657928    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657932    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657936    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.658204 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657973    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657980    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657986    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657993    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.657997    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658002    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658007    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658012    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658016    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658021    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658025    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658030    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658035    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658039    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658044    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658048    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658053    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658057    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658064    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.658988 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658070    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658074    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658079    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658083    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658090    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658094    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658099    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658103    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658107    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658112    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658116    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658121    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658127    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658132    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658137    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658142    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658146    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658150    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658155    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658159    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.659801 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658164    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658168    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658173    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658177    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658181    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658186    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658190    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658195    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658199    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658203    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658207    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658212    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658220    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658223    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658228    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658232    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658236    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658241    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658245    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.660661 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658249    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658253    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658258    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658262    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658267    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658272    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658277    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658281    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658286    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658297    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658301    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658305    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658309    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658313    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658318    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658322    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658326    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658331    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.661336 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658335    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658339    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658344    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.658349    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659738    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659751    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659757    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659769    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659774    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659778    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659783    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659787    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659791    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659795    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659799    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659803    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659807    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659812    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659817    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.661785 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659821    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659842    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659847    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659851    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659855    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659860    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659864    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659868    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659872    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659877    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659881    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659885    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659890    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659894    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659899    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659903    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659907    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659912    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659917    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659921    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.662605 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659929    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659934    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659939    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659944    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659952    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659961    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659968    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659972    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659977    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659982    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659987    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659991    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659995    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.659999    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660004    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660008    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.663119 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660013    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660017    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660022    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660026    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660031    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660035    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660039    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660044    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660048    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660052    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660057    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660061    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660065    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660074    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660079    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660084    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660088    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660093    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660097    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.663543 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660102    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660106    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660110    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660115    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660119    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660124    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660128    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660133    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660137    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660141    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660145    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660149    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660154    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660158    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660162    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660167    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660171    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660175    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660180    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660185    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.664406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660190    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660194    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660198    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660202    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660206    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660211    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660215    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660226    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660231    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660235    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660240    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660244    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.660248    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661930    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661941    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661946    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661954    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661961    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661967    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.665277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661972    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661976    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661980    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661985    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661989    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661994    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.661998    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662002    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662007    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662011    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662015    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662020    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662024    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662028    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662032    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662036    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662041    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662045    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662050    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662055    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.666070 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662059    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662064    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662067    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662071    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662076    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662080    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662085    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662089    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662094    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662098    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662106    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662111    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662115    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662119    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662123    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662128    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662132    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662136    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662140    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662145    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.666592 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662149    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662155    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662161    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662167    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662172    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662176    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662180    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662185    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662189    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662192    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662197    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662202    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662206    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662211    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662216    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662220    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662224    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662228    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662232    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.667341 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662237    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662242    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662246    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662253    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662257    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662261    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662265    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662269    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662274    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662278    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662282    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662286    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662290    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662295    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662299    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662303    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662307    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662311    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662315    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662319    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.667813 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662324    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662327    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662331    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662336    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662340    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662345    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662349    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662353    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662357    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662362    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662365    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662369    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662378    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662382    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662387    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662393    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662397    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662401    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662406    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.668391 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662410    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.662414    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663578    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663594    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663604    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663612    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663619    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663625    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663632    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663639    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663644    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663650    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663656    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663661    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663667    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663672    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663677    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663681    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663686    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663693    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663701    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663705    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663711    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663716    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 14:46:48.668912 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663721    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663729    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663734    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663740    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663745    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663750    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663754    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663759    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663765    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663770    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663776    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663781    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663786    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663791    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663797    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663801    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663809    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663814    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663819    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663824    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663844    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663851    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663856    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663861    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663866    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 14:46:48.669487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663871    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663875    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663881    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663886    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663892    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663896    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663901    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663908    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663914    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663919    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663924    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663929    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663934    2579 flags.go:64] FLAG: --help="false"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663939    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663944    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663949    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663954    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663959    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663965    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663970    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663975    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663979    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663984    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 14:46:48.670103 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663989    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663994    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.663999    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664004    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664008    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664013    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664018    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664023    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664027    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664033    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664038    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664047    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664052    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664058    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664063    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664068    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664074    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664078    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664083    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664091    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664096    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664102    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664107    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664112    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664117    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 14:46:48.670650 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664122    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664127    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664132    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664137    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664149    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664156    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664161    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664166    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664171    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664175    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664180    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664185    2579 flags.go:64] FLAG: --port="10250"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664190    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664195    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-061c918d1d40b3899"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664201    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664205    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664211    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664215    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664230    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664235    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664239    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664250    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664258    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664263    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664268    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664273    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 14:46:48.671284 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664278    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664282    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664288    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664293    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664298    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664302    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664307    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664312    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664317    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664322    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664327    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664332    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664337    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664342    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664347    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664351    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664360    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664365    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664369    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664377    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664381    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664386    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664390    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664395    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664402    2579 flags.go:64] FLAG: --v="2"
Oct 30 14:46:48.671977 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664415    2579 flags.go:64] FLAG: --version="false"
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664422    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664428    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.664435    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664687    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664695    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664701    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664705    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664710    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664715    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664719    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664724    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664728    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664732    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664739    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664745    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664750    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664756    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664760    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664765    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.672617 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664769    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664774    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664778    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664782    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664786    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664791    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664795    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664800    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664804    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664809    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664813    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664819    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664842    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664848    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664852    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664856    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664862    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664867    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664871    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.673113 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664876    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664880    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664884    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664888    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664893    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664897    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664901    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664906    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664909    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664914    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664918    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664922    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664926    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664931    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664935    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664939    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664943    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664952    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664956    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.673568 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664961    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664965    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664969    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664973    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664980    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664986    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664990    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664994    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.664999    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665004    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665009    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665013    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665017    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665022    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665026    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665030    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665034    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665038    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665042    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.674049 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665047    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665051    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665055    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665059    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665063    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665068    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665072    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665077    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665084    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665091    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665096    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665100    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665104    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665108    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665113    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665117    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665121    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665127    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665134    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.674504 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665138    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665142    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665147    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665151    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665155    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665160    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665165    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665169    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665173    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665178    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665182    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665187    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665191    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.665236    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665296    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665315    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665320    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.674969 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665324    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665328    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665334    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665339    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665343    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665347    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665352    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665356    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665360    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665365    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665369    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665373    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665377    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665384    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665389    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665395    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665399    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665404    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665408    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665412    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.675371 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665417    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665422    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665427    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665431    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665435    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665439    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665444    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665448    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665452    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665456    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665461    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665465    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665469    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665474    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665478    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665482    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665486    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665490    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665495    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.675859 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665502    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665508    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665512    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665517    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665521    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665525    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665531    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665535    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665541    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665546    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665550    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665554    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665559    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665564    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665568    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665573    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665577    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665581    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665586    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.676629 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665590    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665594    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665599    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665603    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665608    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665612    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665617    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665621    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665625    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665630    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665635    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665639    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665643    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665647    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665652    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665659    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665664    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665670    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665676    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.677233 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665682    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665686    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665692    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665696    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665700    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665704    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665709    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665713    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665718    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665723    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665728    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665732    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665737    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665740    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665745    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665749    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665754    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665759    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665763    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665767    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.677692 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665771    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665776    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665780    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665784    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665788    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.665794    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.673309    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.673327    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.673344    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.673349    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673408    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673413    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673417    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673421    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673424    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673427    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673430    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673433    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673436    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.678210 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673439    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673444    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673448    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673452    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673455    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673458    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673461    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673464    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673466    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673469    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673472    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673475    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673478    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673480    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673483    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673486    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673489    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673491    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673494    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.678669 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673496    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673501    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673504    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673507    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673510    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673512    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673515    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673518    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673521    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673524    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673526    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673529    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673532    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673535    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673537    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673540    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673542    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673545    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673548    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673550    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.679129 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673553    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673556    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673559    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673561    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673564    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673567    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673570    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673573    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673577    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673579    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673582    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673585    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673588    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673591    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673594    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673596    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673599    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673602    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673604    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673607    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.679590 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673610    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673612    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673615    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673618    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673620    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673623    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673625    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673628    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673631    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673633    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673636    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673638    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673641    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673644    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673646    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673649    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673652    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673655    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673658    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673661    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.680131 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673663    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673666    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673669    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673671    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673674    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673677    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673680    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673685    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673689    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673692    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673695    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673699    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673701    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673705    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673708    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673710    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673713    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.680628 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673715    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.673745    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673916    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673922    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673925    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673928    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673931    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673933    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673936    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673938    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673941    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673944    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673946    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673949    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673952    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.681050 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673956    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673960    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673962    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673965    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673968    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673971    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673974    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673978    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673981    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673984    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673987    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673989    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673992    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673995    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.673997    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674000    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674002    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674005    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674008    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.681406 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674010    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674013    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674015    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674018    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674021    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674024    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674027    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674030    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674033    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674037    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674041    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674044    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674047    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674050    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674053    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674056    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674059    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674061    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674064    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.681894 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674067    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674069    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674072    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674075    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674078    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674080    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674082    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674085    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674088    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674090    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674093    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674096    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674099    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674101    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674104    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674107    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674110    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674113    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674115    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.682352 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674118    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674120    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674123    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674126    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674128    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674131    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674133    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674136    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674139    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674141    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674144    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674147    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674149    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674152    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674154    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674157    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674160    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674163    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674165    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674168    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.682799 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674170    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674173    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674176    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674179    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674181    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674184    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674187    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674189    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674192    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674195    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674200    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674203    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674205    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674208    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674210    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.683277 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.674234    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674267    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674270    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674274    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674277    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674279    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674282    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674284    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674287    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674290    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674295    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674297    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674300    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674303    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674306    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674309    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674312    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674315    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674317    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:46:48.683653 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674320    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674322    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674325    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674328    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674331    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674333    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674336    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674338    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674341    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674344    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674346    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674349    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674351    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674354    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674356    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674359    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674361    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674364    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674366    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:46:48.684269 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674369    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674374    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674377    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674379    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674383    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674387    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674390    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674392    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674395    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674398    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674401    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674403    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674406    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674409    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674411    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674414    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674416    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674419    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674421    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:46:48.684709 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674424    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674426    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674429    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674431    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674434    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674437    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674439    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674442    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674445    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674447    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674450    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674453    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674455    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674459    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674465    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674468    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674471    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674474    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674477    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:46:48.685223 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674479    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674482    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674484    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674487    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674490    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674493    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674496    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674499    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674502    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674504    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674507    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674510    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674512    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674515    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674518    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674521    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674524    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674526    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674529    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674532    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:46:48.685652 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674534    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674537    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674540    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674542    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674545    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674547    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674550    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674554    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674557    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:48.674560    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.674619    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.678216    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.679221    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.679313    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.679346    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.681433    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 14:46:48.686134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.682751    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 14:46:48.694043 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.694025    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 14:46:48.697319 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.697303    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 14:46:48.698147 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.698126    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 14:46:48.706795 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.706763    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa8d8dde-1826-4e04-85e9-4814f3b90aa0:/dev/nvme0n1p3 bc16899c-71fa-4562-a9c4-6b08ac89e97e:/dev/nvme0n1p4]
Oct 30 14:46:48.706906 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.706792    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 14:46:48.712195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.712068    2579 manager.go:217] Machine: {Timestamp:2026-10-30 14:46:48.710698122 +0000 UTC m=+0.362972660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101402 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2220afab71d0e7dee398c485fd5b2b SystemUUID:ec2220af-ab71-d0e7-dee3-98c485fd5b2b BootID:8be41e49-17e9-4e9a-8f14-7bcc2aa83678 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:09:db:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:09:db:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:cd:a4:11:6a:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 14:46:48.712775 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.712762    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 14:46:48.712926 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.712913    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 14:46:48.715585 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715550    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 14:46:48.715728 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715584    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 14:46:48.715803 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715730    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 14:46:48.715803 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715738    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 14:46:48.715803 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715770    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 14:46:48.715803 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.715789    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 14:46:48.716717 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.716701    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 14:46:48.720677 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.720660    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 14:46:48.720805 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.720795    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 14:46:48.721652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.721642    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 14:46:48.721687 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.721656    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 14:46:48.721687 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.721667    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 14:46:48.721687 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.721677    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 14:46:48.721687 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.721685    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 14:46:48.722145 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.722129    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 14:46:48.725965 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.725453    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 14:46:48.730979 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.730961    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 14:46:48.731971 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.731956    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 14:46:48.732041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.731988    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 14:46:48.732168 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732158    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 14:46:48.732201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732173    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 14:46:48.732201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732180    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 14:46:48.732201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732185    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 14:46:48.732201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732191    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 14:46:48.732201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732197    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 14:46:48.732771 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732760    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 14:46:48.732804 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732772    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 14:46:48.732804 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732780    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 14:46:48.732804 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732786    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 14:46:48.732804 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732796    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 14:46:48.732804 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.732805    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 14:46:48.733422 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.733413    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 14:46:48.733457 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.733423    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 14:46:48.737203 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.737187    2579 server.go:1263] "Started kubelet"
Oct 30 14:46:48.737299 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.737269    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 14:46:48.737353 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.737272    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 14:46:48.737407 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.737357    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 14:46:48.738079 ip-10-0-138-143 systemd[1]: Started Kubernetes Kubelet.
Oct 30 14:46:48.739134 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.739042    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 14:46:48.739983 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.739964    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 14:46:48.744462 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.744441    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 14:46:48.745933 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.745916    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 14:46:48.747137 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747122    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 14:46:48.747227 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.747208    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:48.747877 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747857    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 14:46:48.747877 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747859    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 14:46:48.748004 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747888    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 14:46:48.748004 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747972    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 14:46:48.748004 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.747981    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 14:46:48.748765 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.748737    2579 factory.go:55] Registering systemd factory
Oct 30 14:46:48.748765 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.748758    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 14:46:48.749010 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.748992    2579 factory.go:153] Registering CRI-O factory
Oct 30 14:46:48.749010 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.749009    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 14:46:48.749145 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.749051    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 14:46:48.749145 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.749065    2579 factory.go:103] Registering Raw factory
Oct 30 14:46:48.749145 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.749076    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 14:46:48.749956 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.749937    2579 manager.go:319] Starting recovery of all containers
Oct 30 14:46:48.762920 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.762885    2579 manager.go:324] Recovery completed
Oct 30 14:46:48.764948 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.764925    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 14:46:48.768036 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.768021    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.769946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.769908    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.770037 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.769956    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.770037 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.769971    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.770607 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.770592    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 14:46:48.770607 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.770605    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.770709 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.770605    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 14:46:48.770709 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.770665    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 14:46:48.771392 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.771375    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.771456 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.771402    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.771456 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.771412    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.771644 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.771626    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 14:46:48.771730 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.771626    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 14:46:48.772713 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.771622    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-143.ec2.internal.18e355f9f90c9340  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-143.ec2.internal,},FirstTimestamp:2026-10-30 14:46:48.737157952 +0000 UTC m=+0.389432489,LastTimestamp:2026-10-30 14:46:48.737157952 +0000 UTC m=+0.389432489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-143.ec2.internal,}"
Oct 30 14:46:48.773052 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.773040    2579 policy_none.go:50] "Start"
Oct 30 14:46:48.773095 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.773059    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 14:46:48.773095 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.773072    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 14:46:48.775093 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.775081    2579 policy_none.go:44] "Start"
Oct 30 14:46:48.787680 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.787652    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 14:46:48.787680 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.787663    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 14:46:48.787680 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.787652    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 14:46:48.787869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.787607    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-143.ec2.internal.18e355f9fb00c144  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-143.ec2.internal,},FirstTimestamp:2026-10-30 14:46:48.769937732 +0000 UTC m=+0.422212280,LastTimestamp:2026-10-30 14:46:48.769937732 +0000 UTC m=+0.422212280,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-143.ec2.internal,}"
Oct 30 14:46:48.806538 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.806515    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m29rt"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.814556    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.817930    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m29rt"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.819958    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.819992    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820004    2579 server.go:87] "Starting device plugin registration server"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820303    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820317    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820411    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820503    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.820511    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.821845    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.821892    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.825543    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.826824    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.826866    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.826888    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.826928    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 14:46:48.830654 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.827531    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-143.ec2.internal.18e355f9fb0125db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-143.ec2.internal,},FirstTimestamp:2026-10-30 14:46:48.769963483 +0000 UTC m=+0.422238023,LastTimestamp:2026-10-30 14:46:48.769963483 +0000 UTC m=+0.422238023,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-143.ec2.internal,}"
Oct 30 14:46:48.877045 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.877017    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:46:48.921432 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.921350    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.922844 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.922807    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.922939 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.922868    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.922939 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.922883    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.922939 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.922909    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.927960 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.927935    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"]
Oct 30 14:46:48.928004 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.928000    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.928823 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.928798    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.928823 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.928845    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.928982 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.928862    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.930982 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.930968    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.931139 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.931182 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931157    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.931781 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931766    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.931781 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931777    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.931946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931797    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.931946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931800    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.931946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931809    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.931946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.931840    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.934017 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.934001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.934106 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.934031    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:48.934924 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.934908    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:48.935017 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.934929    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:48.935017 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.934938    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:48.945401 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.945387    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.945461 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.945408    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-143.ec2.internal\": node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:48.949201 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.949185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.949249 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.949209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.949249 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:48.949230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.963754 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.963735    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-143.ec2.internal\" not found" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.967561 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.967540    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-143.ec2.internal\" not found" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:46:48.996613 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:48.996582    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.050120 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.049934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.050257 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.050127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.050257 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.050158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.050257 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.050017    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.050257 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.050198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.050257 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.050186    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.097561 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.097530    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.198704 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.198598    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.268603 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.268565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.271720 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.271694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:49.300135 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.300050    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.400918 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.400860    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.502986 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.502898    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.603807 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.603766    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.678979 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.678941    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 14:46:49.679498 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.679125    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 14:46:49.704562 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.704531    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.775490 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.775415    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:46:49.805502 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.805462    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.823985 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.823943    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:41:49 +0000 UTC" deadline="2028-05-05 11:05:13.979116561 +0000 UTC"
Oct 30 14:46:49.824062 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.823986    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13268h18m24.155133955s"
Oct 30 14:46:49.835633 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:49.835600    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca77c311626526aba5ca36ce028bef40.slice/crio-16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20 WatchSource:0}: Error finding container 16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20: Status 404 returned error can't find the container with id 16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20
Oct 30 14:46:49.839188 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:49.839162    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6501707c434eeb14bc7e15cb32ecb7b.slice/crio-6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71 WatchSource:0}: Error finding container 6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71: Status 404 returned error can't find the container with id 6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71
Oct 30 14:46:49.843844 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.843812    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:46:49.905712 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:49.905669    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:49.944817 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.944783    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:46:49.946234 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.946213    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:46:49.946356 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.946252    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:46:49.946356 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:49.946267    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:46:50.005874 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.005825    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:50.106405 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.106305    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:50.207538 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.207503    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:50.308198 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.308157    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:46:50.356578 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.356493    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:46:50.413167 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.413138    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:46:50.447629 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.447583    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:50.472543 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.472515    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:46:50.473877 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.473851    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:46:50.512990 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.512962    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:46:50.723413 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.723326    2579 apiserver.go:52] "Watching apiserver"
Oct 30 14:46:50.747230 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.747193    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 14:46:50.753369 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.753337    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 14:46:50.753787 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.753761    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9vztw","kube-system/global-pull-secret-syncer-pnhqm","kube-system/konnectivity-agent-k4c5w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd","openshift-cluster-node-tuning-operator/tuned-tp4ll","openshift-dns/node-resolver-vnzrb","openshift-image-registry/node-ca-96lz7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal","openshift-multus/multus-rf9g5","openshift-network-operator/iptables-alerter-76w97","openshift-ovn-kubernetes/ovnkube-node-kd44p","openshift-multus/network-metrics-daemon-grqw8","openshift-network-diagnostics/network-check-target-zxt6d","kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"]
Oct 30 14:46:50.756988 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.756961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.758944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.758913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:50.759062 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.759029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:46:50.761328 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.761307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.764731 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.764490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.767458 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.767218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.769703 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.769677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.774216 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.772560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.774216 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.773915    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\""
Oct 30 14:46:50.774678 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.774655    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.774767 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.774746    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 14:46:50.774873 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.774858    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.775618 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.775593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.778728 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.778701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.778833 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.778801    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:46:50.781649 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.781628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.784964 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.784944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.786931 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.786910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:50.787036 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.786987    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:46:50.822545 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.822516    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 14:46:50.822662 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.822609    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zjtwb"
Oct 30 14:46:50.833137 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.833045    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal" event={"ID":"d6501707c434eeb14bc7e15cb32ecb7b","Type":"ContainerStarted","Data":"6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71"}
Oct 30 14:46:50.835196 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.835169    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerStarted","Data":"16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20"}
Oct 30 14:46:50.849348 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.849321    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 14:46:50.861884 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.861838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvm2z\" (UniqueName: \"kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.861899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.861925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.861950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.861984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gzn\" (UniqueName: \"kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.862363 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbr89\" (UniqueName: \"kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdx7\" (UniqueName: \"kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863026 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqqn\" (UniqueName: \"kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt98x\" (UniqueName: \"kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.862999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.863839 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wft\" (UniqueName: \"kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.863975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.864652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2zt\" (UniqueName: \"kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.865463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.864401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9nn\" (UniqueName: \"kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.882747 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.882709    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\""
Oct 30 14:46:50.882981 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.882794    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 14:46:50.882981 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.882946    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 14:46:50.883440 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883420    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\""
Oct 30 14:46:50.883580 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883540    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\""
Oct 30 14:46:50.883658 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883622    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\""
Oct 30 14:46:50.883748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883731    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\""
Oct 30 14:46:50.883867 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883851    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\""
Oct 30 14:46:50.883973 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.883960    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\""
Oct 30 14:46:50.884037 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884025    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 14:46:50.884569 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884550    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 14:46:50.884686 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884603    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 14:46:50.884746 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884697    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\""
Oct 30 14:46:50.884806 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884784    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.884955 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884870    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.885029 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.884996    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 14:46:50.885087 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885042    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 14:46:50.885087 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885080    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 14:46:50.885283 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885249    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.885352 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885286    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.885418 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885375    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 14:46:50.885418 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885402    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 14:46:50.885528 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885440    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.885584 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885531    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.885639 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885619    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.885698 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885688    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.885832 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885816    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 14:46:50.885952 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.885938    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.886021 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.886013    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.886114 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.886100    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 14:46:50.886185 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.886171    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 14:46:50.886309 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.886285    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965490    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.965902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2zt\" (UniqueName: \"kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9nn\" (UniqueName: \"kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.965986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm2z\" (UniqueName: \"kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.966902 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966230    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gzn\" (UniqueName: \"kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966473    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.967769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966521    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966683    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbr89\" (UniqueName: \"kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdx7\" (UniqueName: \"kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966843    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 14:46:50.968641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966916    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966987    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967006    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967018    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966817    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967114    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967170    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967214    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967264    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967269    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.969542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967390    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967687    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967741    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.967831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968016    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968155    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968825    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968922    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.970271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.966967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.968978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969027    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969071    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969079    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqqn\" (UniqueName: \"kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969131    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.969163    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt98x\" (UniqueName: \"kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.971217 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wft\" (UniqueName: \"kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.969534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:46:51.469500068 +0000 UTC m=+3.121774593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969581    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969624    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969871    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.969978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.971996 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970540    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970659    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970759    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.970762    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.970761    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:50.970825    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:46:51.470806269 +0000 UTC m=+3.123080817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.971130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.971304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.971334    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.971417    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.971638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.972616 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.972089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:50.973587 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.973507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:50.980798 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.980773    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:50.997748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.997705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdx7\" (UniqueName: \"kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:50.998643 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.998596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm2z\" (UniqueName: \"kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:50.999011 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:50.998959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9nn\" (UniqueName: \"kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:51.000427 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.000397    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:46:51.000627 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.000433    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:46:51.000627 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.000451    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:51.000627 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.000521    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:46:51.500500313 +0000 UTC m=+3.152774886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:51.002206 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.002179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqqn\" (UniqueName: \"kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:51.003438 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.003394    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gzn\" (UniqueName: \"kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:51.004452 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.004405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wft\" (UniqueName: \"kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:51.006471 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.006446    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbr89\" (UniqueName: \"kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:51.007025 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.006982    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt98x\" (UniqueName: \"kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:51.008534 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.008487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2zt\" (UniqueName: \"kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:51.071366 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.071102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:46:51.078825 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.078790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:51.086761 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.086713    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cce1e62_9cf4_44b6_8f63_9dd948c5f0b4.slice/crio-adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977 WatchSource:0}: Error finding container adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977: Status 404 returned error can't find the container with id adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977
Oct 30 14:46:51.091073 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.091040    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c8e73f_c556_4dfe_b3b8_8b3e60dc98d3.slice/crio-f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b WatchSource:0}: Error finding container f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b: Status 404 returned error can't find the container with id f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b
Oct 30 14:46:51.094611 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.094588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:46:51.095454 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.095393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:46:51.103353 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.103318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:46:51.106619 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.106578    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0badeea4_6775_4cf7_ad74_887fa9ee375a.slice/crio-7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef WatchSource:0}: Error finding container 7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef: Status 404 returned error can't find the container with id 7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef
Oct 30 14:46:51.111086 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.111057    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79986431_390d_4313_af30_b2ff62e0ce68.slice/crio-ecb535529e244b87a3d42ca021088247e62576abe7b1b97c1b605950387c88db WatchSource:0}: Error finding container ecb535529e244b87a3d42ca021088247e62576abe7b1b97c1b605950387c88db: Status 404 returned error can't find the container with id ecb535529e244b87a3d42ca021088247e62576abe7b1b97c1b605950387c88db
Oct 30 14:46:51.111174 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.111151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rf9g5"
Oct 30 14:46:51.114958 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.114931    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d22383_d225_4613_baf3_30a0b288519e.slice/crio-d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779 WatchSource:0}: Error finding container d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779: Status 404 returned error can't find the container with id d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779
Oct 30 14:46:51.122905 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.122855    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee6e3a3_595c_4f16_a095_4e79aae97230.slice/crio-f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc WatchSource:0}: Error finding container f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc: Status 404 returned error can't find the container with id f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc
Oct 30 14:46:51.137756 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.137718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:46:51.139971 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.139935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:46:51.152772 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.152728    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9526eb6d_d032_49ab_8a4e_901e901681a7.slice/crio-dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061 WatchSource:0}: Error finding container dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061: Status 404 returned error can't find the container with id dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061
Oct 30 14:46:51.153723 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:46:51.153680    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d35a9c_d95f_4eba_a095_1f95b931f16a.slice/crio-566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac WatchSource:0}: Error finding container 566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac: Status 404 returned error can't find the container with id 566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac
Oct 30 14:46:51.158172 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.158151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:46:51.195605 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.195515    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:46:51.474944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.474913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:51.475057 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.474953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:51.475133 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.475109    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:51.475185 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.475154    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:51.475246 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.475195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:46:52.475173401 +0000 UTC m=+4.127447948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:51.475246 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.475221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:46:52.475202886 +0000 UTC m=+4.127477432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:51.575755 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.575471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:51.575957 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.575934    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:46:51.576021 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.575964    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:46:51.576021 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.575977    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:51.576134 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:51.576048    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:46:52.576025522 +0000 UTC m=+4.228300074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:51.831541 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.831230    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:41:51 +0000 UTC" deadline="2028-08-07 14:35:18.869200256 +0000 UTC"
Oct 30 14:46:51.831541 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.831366    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15527h48m27.037838924s"
Oct 30 14:46:51.849097 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.849058    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96lz7" event={"ID":"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4","Type":"ContainerStarted","Data":"adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977"}
Oct 30 14:46:51.859601 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.859485    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca77c311626526aba5ca36ce028bef40" containerID="f7ba4e9ca45a549b4080db65efd14be834c0d9839c877510fd1974663e21cbdb" exitCode=0
Oct 30 14:46:51.859601 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.859535    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerDied","Data":"f7ba4e9ca45a549b4080db65efd14be834c0d9839c877510fd1974663e21cbdb"}
Oct 30 14:46:51.867304 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.867263    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"ea24a14a7fda5d85849c60ef8989e4bfd07efd5a8e525889ced07262234c71a3"}
Oct 30 14:46:51.881720 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.881606    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76w97" event={"ID":"71d35a9c-d95f-4eba-a095-1f95b931f16a","Type":"ContainerStarted","Data":"566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac"}
Oct 30 14:46:51.896391 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.896231    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rf9g5" event={"ID":"dee6e3a3-595c-4f16-a095-4e79aae97230","Type":"ContainerStarted","Data":"f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc"}
Oct 30 14:46:51.914600 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.914512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tp4ll" event={"ID":"79986431-390d-4313-af30-b2ff62e0ce68","Type":"ContainerStarted","Data":"ecb535529e244b87a3d42ca021088247e62576abe7b1b97c1b605950387c88db"}
Oct 30 14:46:51.947667 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.947566    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd" event={"ID":"0badeea4-6775-4cf7-ad74-887fa9ee375a","Type":"ContainerStarted","Data":"7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef"}
Oct 30 14:46:51.965423 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.965321    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4c5w" event={"ID":"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3","Type":"ContainerStarted","Data":"f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b"}
Oct 30 14:46:51.979694 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:51.979650    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal" event={"ID":"d6501707c434eeb14bc7e15cb32ecb7b","Type":"ContainerStarted","Data":"2d05b80b7af45e8a4de302278835f1f86b4aa8d30082d3f1f303f812301ca2bb"}
Oct 30 14:46:52.008532 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.008408    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerStarted","Data":"dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061"}
Oct 30 14:46:52.019498 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.019457    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vnzrb" event={"ID":"43d22383-d225-4613-baf3-30a0b288519e","Type":"ContainerStarted","Data":"d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779"}
Oct 30 14:46:52.023715 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.023651    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal" podStartSLOduration=2.023630875 podStartE2EDuration="2.023630875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:52.022538384 +0000 UTC m=+3.674812935" watchObservedRunningTime="2026-10-30 14:46:52.023630875 +0000 UTC m=+3.675905426"
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.483369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.483424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.483572    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.483650    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:46:54.483629443 +0000 UTC m=+6.135903990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.483573    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:52.484135 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.484057    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:46:54.48403824 +0000 UTC m=+6.136312768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:52.584361 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.584325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:52.584550 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.584535    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:46:52.584615 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.584552    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:46:52.584615 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.584561    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:52.584615 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.584606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:46:54.584593134 +0000 UTC m=+6.236867660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.829527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.829659    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.830014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.830022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.830106    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:46:52.830469 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:52.830169    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:46:52.831991 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.831938    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:41:51 +0000 UTC" deadline="2028-04-11 07:53:18.714380119 +0000 UTC"
Oct 30 14:46:52.831991 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:52.831978    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12689h6m25.882407286s"
Oct 30 14:46:53.043794 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:53.043704    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerStarted","Data":"97766a9df919711e30ba24246fcbd5d2c01cfa7c75bfe926fc0493728ef67ba3"}
Oct 30 14:46:53.087346 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:53.087233    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" podStartSLOduration=3.087202875 podStartE2EDuration="3.087202875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:53.086373584 +0000 UTC m=+4.738648134" watchObservedRunningTime="2026-10-30 14:46:53.087202875 +0000 UTC m=+4.739477429"
Oct 30 14:46:54.500770 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.500731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:54.501275 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.500785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:54.501275 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.500963    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:54.501275 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.501034    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:46:58.501015045 +0000 UTC m=+10.153289583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:54.501443 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.501414    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:54.501490 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.501460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:46:58.501445241 +0000 UTC m=+10.153719771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:54.602253 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.602174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:54.602483 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.602461    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:46:54.602566 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.602491    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:46:54.602566 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.602504    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:54.602662 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.602570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:46:58.602552751 +0000 UTC m=+10.254827299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:54.830369 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.830331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:54.830530 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.830468    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:46:54.830938 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.830900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:54.831055 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.830998    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:46:54.831111 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:54.831064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:54.831169 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:54.831143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:46:55.053714 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:55.053090    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="48b1ce7fc4deaeb7097ad571f5c537773a92db9199bfe6433a070c3c6c288c10" exitCode=0
Oct 30 14:46:55.053714 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:55.053143    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"48b1ce7fc4deaeb7097ad571f5c537773a92db9199bfe6433a070c3c6c288c10"}
Oct 30 14:46:56.061191 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.061152    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4c5w" event={"ID":"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3","Type":"ContainerStarted","Data":"2ddb5dc145b0310e963ce33ef1e0e0408cb4900e5e9df83503ce90d7ad2dfd39"}
Oct 30 14:46:56.227156 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.227070    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:56.227776 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.227618    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:46:56.267738 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.267671    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k4c5w" podStartSLOduration=3.114885091 podStartE2EDuration="7.267648254s" totalImagesPullingTime="4.152763163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:56.10309507 +0000 UTC m=+7.755369617" watchObservedRunningTime="2026-10-30 14:46:56.267648254 +0000 UTC m=+7.919922799"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.827944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:56.828077    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.827944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:56.828214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:56.828263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:56.828486 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:56.828333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:46:57.073282 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:57.073241    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96lz7" event={"ID":"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4","Type":"ContainerStarted","Data":"c4cdd5fdcc5fe62a61a5aca89516c41c94929414e08f8e9b8a5758b4d9f3783e"}
Oct 30 14:46:57.113281 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:57.113188    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-96lz7" podStartSLOduration=2.924327402 podStartE2EDuration="8.113173516s" totalImagesPullingTime="5.188846114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:57.11286879 +0000 UTC m=+8.765143334" watchObservedRunningTime="2026-10-30 14:46:57.113173516 +0000 UTC m=+8.765448100"
Oct 30 14:46:58.085196 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.084669    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd" event={"ID":"0badeea4-6775-4cf7-ad74-887fa9ee375a","Type":"ContainerStarted","Data":"36fcee5f5e21e5f2802621b6cb73a58ff939ef7878778604bb813388a0f45cf8"}
Oct 30 14:46:58.085196 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.084814    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:46:58.532727 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.532646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:58.532727 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.532698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:58.532946 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.532766    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:58.532946 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.532902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.532846343 +0000 UTC m=+18.185120873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:46:58.532946 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.532910    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:58.533097 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.532974    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.532957212 +0000 UTC m=+18.185231750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:46:58.633639 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.633599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:58.633794 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.633776    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:46:58.633844 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.633799    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:46:58.633844 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.633812    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:58.633913 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.633880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.633863819 +0000 UTC m=+18.286138347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:46:58.737642 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.737228    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 14:46:58.833888 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.825862    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:46:58.737246983Z","UUID":"b04ecc00-7f4f-4296-8b54-9a12f8b7f7bc","Handler":null,"Name":"","Endpoint":""}
Oct 30 14:46:58.833888 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.830553    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 14:46:58.833888 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.830581    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 14:46:58.835813 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.834497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:46:58.835813 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.834630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:46:58.835813 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.834727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:46:58.835813 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.834802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:46:58.836097 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:58.836075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:46:58.836213 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:46:58.836195    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:46:59.095068 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:59.095027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd" event={"ID":"0badeea4-6775-4cf7-ad74-887fa9ee375a","Type":"ContainerStarted","Data":"8f3ad064e8da814b6d74dd989965a6e6a8b6713b88f14760430da9f1b97c1b07"}
Oct 30 14:46:59.097891 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:46:59.097860    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rf9g5" event={"ID":"dee6e3a3-595c-4f16-a095-4e79aae97230","Type":"ContainerStarted","Data":"a92f7e65a16a9babbb0d5c710a9d13c2fe7ea917232f522a91c17fb7a6e87e65"}
Oct 30 14:47:00.103498 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.103197    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd" event={"ID":"0badeea4-6775-4cf7-ad74-887fa9ee375a","Type":"ContainerStarted","Data":"d0199180fd7749e407e588973cd3d29973aff7a353091f05e8c9496ce24589eb"}
Oct 30 14:47:00.105812 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.105782    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="6d3c49571aa39bb2c73d5dea20658bdfaf2ee140a524f8dc859a5464b3526e12" exitCode=0
Oct 30 14:47:00.105949 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.105881    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"6d3c49571aa39bb2c73d5dea20658bdfaf2ee140a524f8dc859a5464b3526e12"}
Oct 30 14:47:00.107980 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.107895    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vnzrb" event={"ID":"43d22383-d225-4613-baf3-30a0b288519e","Type":"ContainerStarted","Data":"e91053b2dd99e35b24bde303a1d2c6b297c70ac5fc5f37d0a138b4d6bd281f08"}
Oct 30 14:47:00.222323 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.222263    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd" podStartSLOduration=2.891313809 podStartE2EDuration="11.222248864s" totalImagesPullingTime="8.330935055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:00.221914933 +0000 UTC m=+11.874189480" watchObservedRunningTime="2026-10-30 14:47:00.222248864 +0000 UTC m=+11.874523463"
Oct 30 14:47:00.222575 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.222547    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rf9g5" podStartSLOduration=3.598874495 podStartE2EDuration="11.222541186s" totalImagesPullingTime="7.623666691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:59.142697852 +0000 UTC m=+10.794972398" watchObservedRunningTime="2026-10-30 14:47:00.222541186 +0000 UTC m=+11.874815731"
Oct 30 14:47:00.269099 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.269037    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vnzrb" podStartSLOduration=3.428959358 podStartE2EDuration="11.269018023s" totalImagesPullingTime="7.840058665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:00.268598442 +0000 UTC m=+11.920872989" watchObservedRunningTime="2026-10-30 14:47:00.269018023 +0000 UTC m=+11.921292571"
Oct 30 14:47:00.827247 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.827216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:00.827426 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.827285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:00.827426 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:00.827287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:00.827426 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:00.827403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:00.827888 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:00.827861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:00.827988 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:00.827958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:01.111080 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:01.110998    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76w97" event={"ID":"71d35a9c-d95f-4eba-a095-1f95b931f16a","Type":"ContainerStarted","Data":"5a9adc3a7b1f4e68fe95a394bd216d367ac79f9fca7d90d197265f08a818982a"}
Oct 30 14:47:01.150409 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:01.150352    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-76w97" podStartSLOduration=4.08992415 podStartE2EDuration="12.150337836s" totalImagesPullingTime="8.060413686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:01.150227797 +0000 UTC m=+12.802502342" watchObservedRunningTime="2026-10-30 14:47:01.150337836 +0000 UTC m=+12.802612382"
Oct 30 14:47:02.116192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.116152    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tp4ll" event={"ID":"79986431-390d-4313-af30-b2ff62e0ce68","Type":"ContainerStarted","Data":"384f8ae0c85f00ff9e9efe2d7e39a6f28febd6d37e537c9634714d10c0309604"}
Oct 30 14:47:02.118504 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.118472    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="b7164e6e047b41f69081a04f24c59aa55b37468fcd1262d583432dfa96fed2d8" exitCode=0
Oct 30 14:47:02.118652 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.118550    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"b7164e6e047b41f69081a04f24c59aa55b37468fcd1262d583432dfa96fed2d8"}
Oct 30 14:47:02.158747 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.158685    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tp4ll" podStartSLOduration=2.850935076 podStartE2EDuration="13.158671807s" totalImagesPullingTime="10.307736731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:02.158497232 +0000 UTC m=+13.810771791" watchObservedRunningTime="2026-10-30 14:47:02.158671807 +0000 UTC m=+13.810946352"
Oct 30 14:47:02.828218 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.828183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:02.828218 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.828215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:02.828437 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:02.828301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:02.828437 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:02.828391    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:02.828546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:02.828445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:02.828592 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:02.828559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:03.124012 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:03.123933    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="bbd7b01f9bcaf8230157dc6a0b3951df5ff92c60cfd0d94b7ff4a0d941c55b70" exitCode=0
Oct 30 14:47:03.124383 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:03.124019    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"bbd7b01f9bcaf8230157dc6a0b3951df5ff92c60cfd0d94b7ff4a0d941c55b70"}
Oct 30 14:47:04.827920 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:04.827668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:04.828462 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:04.828024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:04.828462 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:04.827699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:04.828462 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:04.828095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:04.828462 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:04.827668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:04.828462 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:04.828170    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:05.131019 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:05.130987    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerStarted","Data":"7aedb05499d9b2fd4eb81900703b20e286d7f8d2fbc5aaf43c47b8e1f049c817"}
Oct 30 14:47:06.135777 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.135538    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="7aedb05499d9b2fd4eb81900703b20e286d7f8d2fbc5aaf43c47b8e1f049c817" exitCode=0
Oct 30 14:47:06.135777 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.135618    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"7aedb05499d9b2fd4eb81900703b20e286d7f8d2fbc5aaf43c47b8e1f049c817"}
Oct 30 14:47:06.140769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.140737    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"71d353e8cedefa6baaecc9cbe066dd1a5e90c3141d74deabeb5e0b3773f00242"}
Oct 30 14:47:06.140890 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.140849    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"0d17a07b4a49b6f83d69bebcd323c81e2119e8d7bc4a1a1a4da10f844afd4c5f"}
Oct 30 14:47:06.140939 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.140912    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"e9c9aca27cb8a242d036002516af9e79b1956f959b171ea4e378ee240635d1e9"}
Oct 30 14:47:06.140975 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.140961    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"d0dc33d68c78a6cb976fff0bb1e32c6afed7bef34c09944dc4eaa091d4d21896"}
Oct 30 14:47:06.141019 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.141005    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"9a420c2de580857d1a09cf36c0ce7080adbc5c165495de8f2139cd3d7af4600e"}
Oct 30 14:47:06.141067 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.141053    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"d32777fe5edaf65c820488e66c05915aa9885a6b2fa4cc90be02c4139922ae7d"}
Oct 30 14:47:06.593978 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.593882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:06.593978 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.593922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:06.594157 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.594036    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 14:47:06.594157 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.594082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret podName:9a72ab8d-d509-46b8-809f-55a708439ffd nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.594070927 +0000 UTC m=+34.246345451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret") pod "global-pull-secret-syncer-pnhqm" (UID: "9a72ab8d-d509-46b8-809f-55a708439ffd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 14:47:06.594157 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.594036    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:47:06.594248 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.594162    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.59414831 +0000 UTC m=+34.246422834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 14:47:06.694701 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.694653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:06.694869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.694818    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 14:47:06.694869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.694851    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 14:47:06.694869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.694861    2579 projected.go:196] Error preparing data for projected volume kube-api-access-2kgzb for pod openshift-network-diagnostics/network-check-target-zxt6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:47:06.694977 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.694909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb podName:b12b5bea-cee8-4db4-89a3-ddec80c8f82a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.694896654 +0000 UTC m=+34.347171179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kgzb" (UniqueName: "kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb") pod "network-check-target-zxt6d" (UID: "b12b5bea-cee8-4db4-89a3-ddec80c8f82a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 14:47:06.827798 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.827752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:06.827798 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.827778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:06.827798 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:06.827805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:06.828042 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.827903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:06.828042 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.828019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:06.828118 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:06.828101    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:07.146075 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:07.146037    2579 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="dfda339a6dc54c7a9eddad94a8c367c1d67550b5b8f1ea3da49198facc9fe572" exitCode=0
Oct 30 14:47:07.146595 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:07.146089    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerDied","Data":"dfda339a6dc54c7a9eddad94a8c367c1d67550b5b8f1ea3da49198facc9fe572"}
Oct 30 14:47:08.151488 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.151391    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"2aa658e6f98333970391ca61fca06ca13ea1dccd8042b3b456112544d7aa5613"}
Oct 30 14:47:08.154564 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.154532    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vztw" event={"ID":"9526eb6d-d032-49ab-8a4e-901e901681a7","Type":"ContainerStarted","Data":"4a380af4d4a96598b7e750eff7451b17a888753406bf82e66e68fba0e3681970"}
Oct 30 14:47:08.225936 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.225879    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9vztw" podStartSLOduration=8.105552387 podStartE2EDuration="19.22586457s" totalImagesPullingTime="11.120312183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:08.21329656 +0000 UTC m=+19.865571107" watchObservedRunningTime="2026-10-30 14:47:08.22586457 +0000 UTC m=+19.878139116"
Oct 30 14:47:08.828254 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.828221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:08.828400 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.828323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:08.828400 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:08.828366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:08.828400 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:08.828359    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:08.828514 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:08.828446    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:08.828570 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:08.828550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:10.163067 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.163027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" event={"ID":"ea952af6-f7f1-43d6-82b4-8137b837a4e0","Type":"ContainerStarted","Data":"69189f921cbba3cfd9cc3ff12c3787e6eace4a550546e89c856596994895e9e0"}
Oct 30 14:47:10.163668 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.163356    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:10.179023 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.178996    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:10.252774 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.252634    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p" podStartSLOduration=6.893665315 podStartE2EDuration="21.252622163s" totalImagesPullingTime="14.358956848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:10.252143254 +0000 UTC m=+21.904417799" watchObservedRunningTime="2026-10-30 14:47:10.252622163 +0000 UTC m=+21.904896745"
Oct 30 14:47:10.828105 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.828072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:10.828105 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.828101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:10.828279 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:10.828172    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:10.828279 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:10.828220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:10.828362 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:10.828315    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:10.828411 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:10.828394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:11.165105 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:11.165034    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:11.165469 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:11.165450    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:11.184641 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:11.184615    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:12.073612 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.073579    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pnhqm"]
Oct 30 14:47:12.073846 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.073722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:12.073846 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:12.073811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:12.079509 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.079486    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxt6d"]
Oct 30 14:47:12.079611 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.079575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:12.079659 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:12.079645    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:12.079939 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.079920    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-grqw8"]
Oct 30 14:47:12.080016 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.080006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:12.080090 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:12.080076    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:12.167622 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:12.167594    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:13.170005 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:13.169977    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:13.827393 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:13.827343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:13.827393 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:13.827389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:13.827662 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:13.827368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:13.827662 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:13.827488    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:13.827662 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:13.827553    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:13.827662 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:13.827605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:14.740920 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:14.740875    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:15.827326 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:15.827287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:15.827326 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:15.827309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:15.827810 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:15.827391    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:15.827810 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:15.827455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:15.827810 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:15.827571    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:15.827810 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:15.827658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:17.828017 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:17.827985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:17.828463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:17.828012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:17.828463 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:17.828085    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnhqm" podUID="9a72ab8d-d509-46b8-809f-55a708439ffd"
Oct 30 14:47:17.828463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:17.828131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:17.828463 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:17.828177    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxt6d" podUID="b12b5bea-cee8-4db4-89a3-ddec80c8f82a"
Oct 30 14:47:17.828463 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:17.828271    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-grqw8" podUID="82e0ad12-29e4-4892-841b-801edb30b567"
Oct 30 14:47:18.624171 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.624144    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeReady"
Oct 30 14:47:18.624327 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.624293    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 14:47:18.671206 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.671176    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tlhw9"]
Oct 30 14:47:18.677491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.677466    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kfr99"]
Oct 30 14:47:18.677623 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.677591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:18.681876 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.681852    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lvqzv"]
Oct 30 14:47:18.682019 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.681912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:18.685239 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.685218    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:47:18.685386 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.685328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:18.688590 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.688568    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"]
Oct 30 14:47:18.689410 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.688798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.693729 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.692742    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:47:18.693729 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.692872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.703477 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.703433    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"]
Oct 30 14:47:18.703868 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.703574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.704937 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.704917    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.706091 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.706075    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.707372 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.707351    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58ddf469d5-lvlsc"]
Oct 30 14:47:18.707627 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.707607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:18.712356 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.712249    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"]
Oct 30 14:47:18.713456 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.713000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.719203 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.718534    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"]
Oct 30 14:47:18.719429 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.719341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.719931 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.719599    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 14:47:18.719931 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.719731    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 14:47:18.719931 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.719814    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 14:47:18.720243 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.720033    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 14:47:18.720243 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.720201    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9fbb\""
Oct 30 14:47:18.721056 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.720897    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 14:47:18.725853 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.724701    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-njl5k"]
Oct 30 14:47:18.725853 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.724921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.729334 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.729119    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"]
Oct 30 14:47:18.729334 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.729286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.732405 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.732387    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"]
Oct 30 14:47:18.732499 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.732461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.736091 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.735386    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"]
Oct 30 14:47:18.736091 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.735945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.740306 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.740287    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"]
Oct 30 14:47:18.740599 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.740580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.743453 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.743433    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9849b"]
Oct 30 14:47:18.743844 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.743814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:18.757035 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.757011    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2fr4t"]
Oct 30 14:47:18.757175 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.757092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.761341 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761323    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tlhw9"]
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761378    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kfr99"]
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761402    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lvqzv"]
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761422    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"]
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761442    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"]
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.761460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761462    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761483    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58ddf469d5-lvlsc"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761504    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761563    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-njl5k"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761588    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761606    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761624    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fr4t"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761643    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761655    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761667    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9849b"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761678    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"]
Oct 30 14:47:18.761748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.761691    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"]
Oct 30 14:47:18.771244 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.771351 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.771351 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.771351 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwp8v\" (UniqueName: \"kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v\") pod \"kas-connection-checker-685895c7c9-kfr99\" (UID: \"26527e26-b484-446f-b704-ee332fea2f2e\") " pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:18.771351 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.771546 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vprm\" (UniqueName: \"kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm\") pod \"volume-data-source-validator-58c9f5f46d-qdfhs\" (UID: \"30c328c3-1bca-4576-9fd0-c669059e423e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.771896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.772213 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.771893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxdr\" (UniqueName: \"kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr\") pod \"kas-connection-checker-685895c7c9-lvqzv\" (UID: \"ebdd075b-3f66-4f77-ab73-e8f080e78f37\") " pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:18.772213 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.772213 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.772213 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpt7\" (UniqueName: \"kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.772388 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.772445 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.772489 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpr6\" (UniqueName: \"kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.772531 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.772614 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln786\" (UniqueName: \"kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.772614 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.772695 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frz62\" (UniqueName: \"kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.772695 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmksh\" (UniqueName: \"kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh\") pod \"kas-connection-checker-685895c7c9-tlhw9\" (UID: \"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8\") " pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:18.772772 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.772772 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.772772 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.772755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.808385 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808360    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.808511 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808391    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808511 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808501    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808615 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808520    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 14:47:18.808615 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808549    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808615 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808548    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.808731 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808675    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808781 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808735    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808853 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808823    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.808917 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808898    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 14:47:18.808917 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808909    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.809009 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808923    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809009 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808968    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809009 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808982    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 14:47:18.809009 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.808998    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809036    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809044    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809077    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809083    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809098    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809119    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809152    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809081    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 14:47:18.809195 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809188    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809492 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809217    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:18.809492 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809230    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809492 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809245    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 14:47:18.809492 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809265    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 14:47:18.809649 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.809562    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 14:47:18.821559 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821538    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 14:47:18.821699 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821676    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 14:47:18.821699 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821566    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 14:47:18.821699 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821626    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821747    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821807    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821824    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8pjkc\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821892    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821905    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l2gbg\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821914    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821938    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 14:47:18.821944 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821944    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-slvwr\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821986    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wbwnx\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821991    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.821940    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822035    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g9j6g\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822043    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bhrs9\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822037    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822101    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5vnjn\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822113    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nrd46\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822181    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\""
Oct 30 14:47:18.822278 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822236    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qts79\""
Oct 30 14:47:18.822762 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822549    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nh5lx\""
Oct 30 14:47:18.822893 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.822869    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 14:47:18.873137 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.873137 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpt7\" (UniqueName: \"kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.873437    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.37341615 +0000 UTC m=+31.025690691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : configmap references non-existent config key: service-ca.crt
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zttn\" (UniqueName: \"kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.873563 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpr6\" (UniqueName: \"kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5k9\" (UniqueName: \"kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9\") pod \"network-check-source-6f78d59f99-k5zrz\" (UID: \"ba2f7212-f693-4d67-8b61-fc75f1ad0372\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln786\" (UniqueName: \"kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh59\" (UniqueName: \"kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frz62\" (UniqueName: \"kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.873953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.873993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vpn\" (UniqueName: \"kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmksh\" (UniqueName: \"kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh\") pod \"kas-connection-checker-685895c7c9-tlhw9\" (UID: \"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8\") " pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874339    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874358    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.874491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.874552    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp8v\" (UniqueName: \"kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v\") pod \"kas-connection-checker-685895c7c9-kfr99\" (UID: \"26527e26-b484-446f-b704-ee332fea2f2e\") " pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzwc\" (UniqueName: \"kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr2p\" (UniqueName: \"kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.874932    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.374912732 +0000 UTC m=+31.027187258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.875022 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.874957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875059    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vprm\" (UniqueName: \"kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm\") pod \"volume-data-source-validator-58c9f5f46d-qdfhs\" (UID: \"30c328c3-1bca-4576-9fd0-c669059e423e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875257    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.875704 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxdr\" (UniqueName: \"kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr\") pod \"kas-connection-checker-685895c7c9-lvqzv\" (UID: \"ebdd075b-3f66-4f77-ab73-e8f080e78f37\") " pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.875413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.875972    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.875988    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b65c48686-qftvr: secret "image-registry-tls" not found
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.876042    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls podName:8407bb01-c8e0-4624-bd80-c27459b6c96e nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.376025835 +0000 UTC m=+31.028300372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls") pod "image-registry-b65c48686-qftvr" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e") : secret "image-registry-tls" not found
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.876115    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.876125    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-766565bc5f-b7l2r: secret "image-registry-tls" not found
Oct 30 14:47:18.876487 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.876157    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls podName:cbee651d-49b7-4013-b903-da36898913cf nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.376147262 +0000 UTC m=+31.028421789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls") pod "image-registry-766565bc5f-b7l2r" (UID: "cbee651d-49b7-4013-b903-da36898913cf") : secret "image-registry-tls" not found
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.877097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.879548    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.879682    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.879742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.379725443 +0000 UTC m=+31.031999983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : secret "router-metrics-certs-default" not found
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.879775    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.879819    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.879852    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.879959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.880139    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.880301    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.880416 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.880342    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.881767 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.881747    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.904559 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.904527    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frz62\" (UniqueName: \"kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:18.904716 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.904615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpr6\" (UniqueName: \"kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:18.905383 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.905359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.906376 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.906356    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmksh\" (UniqueName: \"kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh\") pod \"kas-connection-checker-685895c7c9-tlhw9\" (UID: \"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8\") " pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:18.906460 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.906446    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.908740 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.908718    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpt7\" (UniqueName: \"kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:18.910160 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.910141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:18.912595 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.912567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp8v\" (UniqueName: \"kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v\") pod \"kas-connection-checker-685895c7c9-kfr99\" (UID: \"26527e26-b484-446f-b704-ee332fea2f2e\") " pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:18.912718 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.912702    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln786\" (UniqueName: \"kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:18.916178 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.916154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vprm\" (UniqueName: \"kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm\") pod \"volume-data-source-validator-58c9f5f46d-qdfhs\" (UID: \"30c328c3-1bca-4576-9fd0-c669059e423e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:18.916547 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.916525    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxdr\" (UniqueName: \"kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr\") pod \"kas-connection-checker-685895c7c9-lvqzv\" (UID: \"ebdd075b-3f66-4f77-ab73-e8f080e78f37\") " pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:18.921157 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.921137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:18.976016 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.975988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh59\" (UniqueName: \"kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vpn\" (UniqueName: \"kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.976181 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzwc\" (UniqueName: \"kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr2p\" (UniqueName: \"kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.976253    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.976314    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert podName:bfad0d99-650b-4984-8afb-757f92bf7db9 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.476294214 +0000 UTC m=+31.128568761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m9l7n" (UID: "bfad0d99-650b-4984-8afb-757f92bf7db9") : secret "networking-console-plugin-cert" not found
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976459    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.976542 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zttn\" (UniqueName: \"kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5k9\" (UniqueName: \"kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9\") pod \"network-check-source-6f78d59f99-k5zrz\" (UID: \"ba2f7212-f693-4d67-8b61-fc75f1ad0372\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.976915    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.977058    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.977173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.977258 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.977209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:18.977660 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.977464    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 14:47:18.977660 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.977531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls podName:4ae91844-19fb-4008-89ce-5e427270dc96 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.477513822 +0000 UTC m=+31.129788367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v4588" (UID: "4ae91844-19fb-4008-89ce-5e427270dc96") : secret "samples-operator-tls" not found
Oct 30 14:47:18.977660 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.977589    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:18.977660 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:18.977624    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:19.477613003 +0000 UTC m=+31.129887541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:18.977899 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.977886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.978192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.977978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.978311 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.978286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.978311 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.978297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:18.978447 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.978388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.978967 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.978948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:18.979112 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.979097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:18.979236 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.979222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:18.993104 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:18.993083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:19.002564 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.002538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:19.002698 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.002679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vpn\" (UniqueName: \"kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:19.004459 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.004439    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh59\" (UniqueName: \"kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:19.004580 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.004560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr2p\" (UniqueName: \"kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:19.004819 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.004805    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzwc\" (UniqueName: \"kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:19.013085 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.012960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:19.014715 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.014667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zttn\" (UniqueName: \"kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:19.015316 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.015296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5k9\" (UniqueName: \"kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9\") pod \"network-check-source-6f78d59f99-k5zrz\" (UID: \"ba2f7212-f693-4d67-8b61-fc75f1ad0372\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:19.060583 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.060545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:19.071204 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.071070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:19.078398 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.078369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:19.085731 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.085700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:19.099010 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.098956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:19.111551 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.111526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:19.121041 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.119443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:19.366691 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.366422    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lvqzv"]
Oct 30 14:47:19.370011 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.369161    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kfr99"]
Oct 30 14:47:19.372037 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.372016    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tlhw9"]
Oct 30 14:47:19.386613 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.386540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:19.386613 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.386579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:19.386748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.386627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:19.386748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.386654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:19.386748 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386725    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 14:47:19.386748 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.386733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386785    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.386767094 +0000 UTC m=+32.039041640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : secret "router-metrics-certs-default" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386789    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386807    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b65c48686-qftvr: secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386817    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386877    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.386864423 +0000 UTC m=+32.039138961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386728    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386892    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-766565bc5f-b7l2r: secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls podName:8407bb01-c8e0-4624-bd80-c27459b6c96e nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.386883911 +0000 UTC m=+32.039158436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls") pod "image-registry-b65c48686-qftvr" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e") : secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386922    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls podName:cbee651d-49b7-4013-b903-da36898913cf nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.386912994 +0000 UTC m=+32.039187521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls") pod "image-registry-766565bc5f-b7l2r" (UID: "cbee651d-49b7-4013-b903-da36898913cf") : secret "image-registry-tls" not found
Oct 30 14:47:19.386974 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.386934    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.386927941 +0000 UTC m=+32.039202465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : configmap references non-existent config key: service-ca.crt
Oct 30 14:47:19.407234 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.407200    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"]
Oct 30 14:47:19.407880 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.407861    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"]
Oct 30 14:47:19.414716 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.414686    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb767b7e7_8048_4086_aa1b_5ec54e3d0568.slice/crio-12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486 WatchSource:0}: Error finding container 12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486: Status 404 returned error can't find the container with id 12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486
Oct 30 14:47:19.415332 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.415300    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ec3816_9582_4e08_b9d0_9d03174f5628.slice/crio-d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda WatchSource:0}: Error finding container d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda: Status 404 returned error can't find the container with id d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda
Oct 30 14:47:19.424976 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.424951    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"]
Oct 30 14:47:19.426680 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.426640    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c328c3_1bca_4576_9fd0_c669059e423e.slice/crio-5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858 WatchSource:0}: Error finding container 5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858: Status 404 returned error can't find the container with id 5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858
Oct 30 14:47:19.467541 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.467497    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-njl5k"]
Oct 30 14:47:19.472769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.472739    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"]
Oct 30 14:47:19.480326 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.480294    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107da892_9033_4172_b313_2431ef06d16c.slice/crio-d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4 WatchSource:0}: Error finding container d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4: Status 404 returned error can't find the container with id d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4
Oct 30 14:47:19.480854 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.480526    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2f7212_f693_4d67_8b61_fc75f1ad0372.slice/crio-3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1 WatchSource:0}: Error finding container 3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1: Status 404 returned error can't find the container with id 3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1
Oct 30 14:47:19.486953 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.486931    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"]
Oct 30 14:47:19.487280 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.487255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:19.487396 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.487380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:19.487478 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.487414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:19.487478 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487435    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 14:47:19.487602 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487492    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert podName:bfad0d99-650b-4984-8afb-757f92bf7db9 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.487475872 +0000 UTC m=+32.139750397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m9l7n" (UID: "bfad0d99-650b-4984-8afb-757f92bf7db9") : secret "networking-console-plugin-cert" not found
Oct 30 14:47:19.487602 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487568    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 14:47:19.487708 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487614    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:19.487708 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487622    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls podName:4ae91844-19fb-4008-89ce-5e427270dc96 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.487605926 +0000 UTC m=+32.139880463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v4588" (UID: "4ae91844-19fb-4008-89ce-5e427270dc96") : secret "samples-operator-tls" not found
Oct 30 14:47:19.487708 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:19.487676    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.487661301 +0000 UTC m=+32.139935832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:19.490216 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.490192    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3374e02_805e_4d08_97c9_2207b67e62f8.slice/crio-d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da WatchSource:0}: Error finding container d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da: Status 404 returned error can't find the container with id d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da
Oct 30 14:47:19.528271 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.528250    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9849b"]
Oct 30 14:47:19.530489 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:19.530465    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f9e692_eeb3_4000_80dc_0eebeb13f1db.slice/crio-e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3 WatchSource:0}: Error finding container e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3: Status 404 returned error can't find the container with id e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3
Oct 30 14:47:19.828131 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.828043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:19.828131 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.828061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:19.828310 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.828186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:19.843588 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.843560    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 14:47:19.843825 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.843798    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 14:47:19.843825 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.843809    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\""
Oct 30 14:47:19.843972 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.843926    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\""
Oct 30 14:47:19.843972 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:19.843948    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\""
Oct 30 14:47:20.201252 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.201218    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerStarted","Data":"d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4"}
Oct 30 14:47:20.205357 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.205288    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg" event={"ID":"b767b7e7-8048-4086-aa1b-5ec54e3d0568","Type":"ContainerStarted","Data":"12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486"}
Oct 30 14:47:20.208193 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.208165    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz" event={"ID":"ba2f7212-f693-4d67-8b61-fc75f1ad0372","Type":"ContainerStarted","Data":"3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1"}
Oct 30 14:47:20.209823 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.209795    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs" event={"ID":"30c328c3-1bca-4576-9fd0-c669059e423e","Type":"ContainerStarted","Data":"5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858"}
Oct 30 14:47:20.213374 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.213345    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kfr99" event={"ID":"26527e26-b484-446f-b704-ee332fea2f2e","Type":"ContainerStarted","Data":"84dc4c25904340fd18ce8bab822d59573073225ddf06ba6b53bc0bc26b8db30f"}
Oct 30 14:47:20.213479 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.213394    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kfr99" event={"ID":"26527e26-b484-446f-b704-ee332fea2f2e","Type":"ContainerStarted","Data":"99683769ba873075d3c607cfc699175d3a8f17ce0e13050f6462fafcb1b01161"}
Oct 30 14:47:20.214805 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.214754    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm" event={"ID":"09ec3816-9582-4e08-b9d0-9d03174f5628","Type":"ContainerStarted","Data":"d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda"}
Oct 30 14:47:20.219006 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.218323    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lvqzv" event={"ID":"ebdd075b-3f66-4f77-ab73-e8f080e78f37","Type":"ContainerStarted","Data":"1beb6b2b6f9b59287aec0f1d0e00e0fb1721a18b5cbf974ca146ebbc0d0b50e5"}
Oct 30 14:47:20.219006 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.218366    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lvqzv" event={"ID":"ebdd075b-3f66-4f77-ab73-e8f080e78f37","Type":"ContainerStarted","Data":"03a570dbbc6da03788d6b7bf4235df2949868913df66848c77cc4b05e3c3e149"}
Oct 30 14:47:20.220596 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.220537    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9849b" event={"ID":"b7f9e692-eeb3-4000-80dc-0eebeb13f1db","Type":"ContainerStarted","Data":"e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3"}
Oct 30 14:47:20.222433 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.222405    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" event={"ID":"f3374e02-805e-4d08-97c9-2207b67e62f8","Type":"ContainerStarted","Data":"d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da"}
Oct 30 14:47:20.225744 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.225718    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tlhw9" event={"ID":"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8","Type":"ContainerStarted","Data":"21c6830580b67d032f211cf234980ee7a3ed425734e65d563071bc9740c2e201"}
Oct 30 14:47:20.225865 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.225761    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tlhw9" event={"ID":"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8","Type":"ContainerStarted","Data":"f6d0f2af294b8d5e5a62ebd7753a8a96202c9ef504329f189773253419c56b10"}
Oct 30 14:47:20.297358 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.297274    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kfr99" podStartSLOduration=364.297254527 podStartE2EDuration="6m4.297254527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:20.257100151 +0000 UTC m=+31.909374695" watchObservedRunningTime="2026-10-30 14:47:20.297254527 +0000 UTC m=+31.949529074"
Oct 30 14:47:20.298290 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.298244    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tlhw9" podStartSLOduration=364.298232822 podStartE2EDuration="6m4.298232822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:20.296901581 +0000 UTC m=+31.949176127" watchObservedRunningTime="2026-10-30 14:47:20.298232822 +0000 UTC m=+31.950507368"
Oct 30 14:47:20.396823 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.396789    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:20.397026 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.396902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.396879664 +0000 UTC m=+34.049154193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:20.397344 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.396629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:20.397499 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.397478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:20.397646 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.397631    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:20.397702 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.397650    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-766565bc5f-b7l2r: secret "image-registry-tls" not found
Oct 30 14:47:20.397754 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.397730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:20.397868 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.397853    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 14:47:20.397926 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.397899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.397882297 +0000 UTC m=+34.050156826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : secret "router-metrics-certs-default" not found
Oct 30 14:47:20.397981 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.397948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:20.398028 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.397982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls podName:cbee651d-49b7-4013-b903-da36898913cf nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.397972578 +0000 UTC m=+34.050247106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls") pod "image-registry-766565bc5f-b7l2r" (UID: "cbee651d-49b7-4013-b903-da36898913cf") : secret "image-registry-tls" not found
Oct 30 14:47:20.398028 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.398015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:20.398143 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.398119    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:20.398143 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.398132    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b65c48686-qftvr: secret "image-registry-tls" not found
Oct 30 14:47:20.398238 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.398166    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls podName:8407bb01-c8e0-4624-bd80-c27459b6c96e nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.398155089 +0000 UTC m=+34.050429616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls") pod "image-registry-b65c48686-qftvr" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e") : secret "image-registry-tls" not found
Oct 30 14:47:20.398335 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.398288    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.39828007 +0000 UTC m=+34.050554593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : configmap references non-existent config key: service-ca.crt
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.498792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.498950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.498983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499170    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499231    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.499211937 +0000 UTC m=+34.151486463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499629    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499683    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert podName:bfad0d99-650b-4984-8afb-757f92bf7db9 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.499667792 +0000 UTC m=+34.151942319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m9l7n" (UID: "bfad0d99-650b-4984-8afb-757f92bf7db9") : secret "networking-console-plugin-cert" not found
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499740    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 14:47:20.499869 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:20.499771    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls podName:4ae91844-19fb-4008-89ce-5e427270dc96 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:22.49976102 +0000 UTC m=+34.152035547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v4588" (UID: "4ae91844-19fb-4008-89ce-5e427270dc96") : secret "samples-operator-tls" not found
Oct 30 14:47:20.606958 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.606918    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:20.607153 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.607110    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:20.608385 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.608158    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:20.647972 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:20.647911    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lvqzv" podStartSLOduration=364.647887533 podStartE2EDuration="6m4.647887533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:20.337202049 +0000 UTC m=+31.989476596" watchObservedRunningTime="2026-10-30 14:47:20.647887533 +0000 UTC m=+32.300162080"
Oct 30 14:47:22.238493 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.238410    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs" event={"ID":"30c328c3-1bca-4576-9fd0-c669059e423e","Type":"ContainerStarted","Data":"846e42e5537db167a77834d40fde09d9cf9b800cd68da57ca8101a090cd6eeff"}
Oct 30 14:47:22.419976 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.419934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:22.420135 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.420062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:22.420194 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.420169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:22.420243 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.420205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:22.420292 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.420258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:22.420407 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420391    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:22.420455 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420412    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b65c48686-qftvr: secret "image-registry-tls" not found
Oct 30 14:47:22.420503 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls podName:8407bb01-c8e0-4624-bd80-c27459b6c96e nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.420451794 +0000 UTC m=+38.072726333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls") pod "image-registry-b65c48686-qftvr" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e") : secret "image-registry-tls" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420906    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.420891225 +0000 UTC m=+38.073165763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : configmap references non-existent config key: service-ca.crt
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420968    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420983    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.420966    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.421035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.421017252 +0000 UTC m=+38.073291795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : secret "router-metrics-certs-default" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.421036    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-766565bc5f-b7l2r: secret "image-registry-tls" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.421053    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.421044116 +0000 UTC m=+38.073318646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:22.421116 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.421084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls podName:cbee651d-49b7-4013-b903-da36898913cf nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.421073304 +0000 UTC m=+38.073347851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls") pod "image-registry-766565bc5f-b7l2r" (UID: "cbee651d-49b7-4013-b903-da36898913cf") : secret "image-registry-tls" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.520786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.520854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.521013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521153    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521216    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert podName:bfad0d99-650b-4984-8afb-757f92bf7db9 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.521197943 +0000 UTC m=+38.173472473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m9l7n" (UID: "bfad0d99-650b-4984-8afb-757f92bf7db9") : secret "networking-console-plugin-cert" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521626    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521684    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls podName:4ae91844-19fb-4008-89ce-5e427270dc96 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.521668239 +0000 UTC m=+38.173942777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v4588" (UID: "4ae91844-19fb-4008-89ce-5e427270dc96") : secret "samples-operator-tls" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521741    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:22.521815 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.521772    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.521762092 +0000 UTC m=+38.174036621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:22.621946 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.621901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:22.622127 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.622026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:22.622203 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.622135    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 14:47:22.622203 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:22.622191    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:54.622178205 +0000 UTC m=+66.274452729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : secret "metrics-daemon-secret" not found
Oct 30 14:47:22.625316 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.625284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:22.724244 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.724112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:22.729164 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.729116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:22.868289 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.868247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:22.888468 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:22.888178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:23.203341 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:23.203276    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs" podStartSLOduration=331.854621132 podStartE2EDuration="5m34.203253112s" totalImagesPullingTime="2.34863198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:22.279297773 +0000 UTC m=+33.931572320" watchObservedRunningTime="2026-10-30 14:47:23.203253112 +0000 UTC m=+34.855527664"
Oct 30 14:47:23.204708 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:23.204680    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pnhqm"]
Oct 30 14:47:23.244115 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:23.244039    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pnhqm" event={"ID":"9a72ab8d-d509-46b8-809f-55a708439ffd","Type":"ContainerStarted","Data":"b51d9b655bda7ef0a3ab2e1e5e2d8731a637884388124f31fb796babe4d9535f"}
Oct 30 14:47:23.296125 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:23.296041    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxt6d"]
Oct 30 14:47:23.301467 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:23.301431    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12b5bea_cee8_4db4_89a3_ddec80c8f82a.slice/crio-b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2 WatchSource:0}: Error finding container b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2: Status 404 returned error can't find the container with id b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2
Oct 30 14:47:24.248982 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.248931    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxt6d" event={"ID":"b12b5bea-cee8-4db4-89a3-ddec80c8f82a","Type":"ContainerStarted","Data":"b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2"}
Oct 30 14:47:24.251737 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.251716    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/0.log"
Oct 30 14:47:24.251908 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.251746    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3374e02-805e-4d08-97c9-2207b67e62f8" containerID="77d13eb4c64bd9b1dc82857250edda0d152ef369de3258e769dfaf602cdf7a81" exitCode=255
Oct 30 14:47:24.251908 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.251814    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" event={"ID":"f3374e02-805e-4d08-97c9-2207b67e62f8","Type":"ContainerDied","Data":"77d13eb4c64bd9b1dc82857250edda0d152ef369de3258e769dfaf602cdf7a81"}
Oct 30 14:47:24.252196 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.252180    2579 scope.go:122] "RemoveContainer" containerID="77d13eb4c64bd9b1dc82857250edda0d152ef369de3258e769dfaf602cdf7a81"
Oct 30 14:47:24.262890 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.262821    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg" event={"ID":"b767b7e7-8048-4086-aa1b-5ec54e3d0568","Type":"ContainerStarted","Data":"e6a342fc6f67b0f1c5f4050e1770f9b49e98db48674797f3e599f1c27314ff90"}
Oct 30 14:47:24.338896 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:24.338771    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg" podStartSLOduration=330.701355041 podStartE2EDuration="5m35.33874928s" totalImagesPullingTime="4.637394239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:24.33736059 +0000 UTC m=+35.989635138" watchObservedRunningTime="2026-10-30 14:47:24.33874928 +0000 UTC m=+35.991023827"
Oct 30 14:47:25.271174 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.270530    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm" event={"ID":"09ec3816-9582-4e08-b9d0-9d03174f5628","Type":"ContainerStarted","Data":"e014cf88bfd9b477e3889e3288fa4d67af49ee817b82da43369fc76393f88a94"}
Oct 30 14:47:25.273483 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.273418    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9849b" event={"ID":"b7f9e692-eeb3-4000-80dc-0eebeb13f1db","Type":"ContainerStarted","Data":"fb1b297d1e1161b4937406dddb724d75cc41604cfed4d0176fff50a8302d1282"}
Oct 30 14:47:25.277322 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.277297    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:25.278159 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.278135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/0.log"
Oct 30 14:47:25.278261 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.278178    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3374e02-805e-4d08-97c9-2207b67e62f8" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74" exitCode=255
Oct 30 14:47:25.278315 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.278264    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" event={"ID":"f3374e02-805e-4d08-97c9-2207b67e62f8","Type":"ContainerDied","Data":"2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"}
Oct 30 14:47:25.278361 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.278316    2579 scope.go:122] "RemoveContainer" containerID="77d13eb4c64bd9b1dc82857250edda0d152ef369de3258e769dfaf602cdf7a81"
Oct 30 14:47:25.278736 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.278718    2579 scope.go:122] "RemoveContainer" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"
Oct 30 14:47:25.279211 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:25.278951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7j8cn_openshift-console-operator(f3374e02-805e-4d08-97c9-2207b67e62f8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" podUID="f3374e02-805e-4d08-97c9-2207b67e62f8"
Oct 30 14:47:25.283565 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.283543    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerStarted","Data":"856c8dca650f9f0197a0c3b97999e1cc9ff1541f48a65a77a8cf43985995eaf7"}
Oct 30 14:47:25.318926 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.318868    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm" podStartSLOduration=331.151637198 podStartE2EDuration="5m36.318816712s" totalImagesPullingTime="5.167179514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:25.3184958 +0000 UTC m=+36.970770347" watchObservedRunningTime="2026-10-30 14:47:25.318816712 +0000 UTC m=+36.971091258"
Oct 30 14:47:25.364922 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.364258    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9849b" podStartSLOduration=1.944695146 podStartE2EDuration="7.364242477s" totalImagesPullingTime="5.419547331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:25.362463538 +0000 UTC m=+37.014738085" watchObservedRunningTime="2026-10-30 14:47:25.364242477 +0000 UTC m=+37.016517017"
Oct 30 14:47:25.454686 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.454618    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-njl5k" podStartSLOduration=331.416149625 podStartE2EDuration="5m36.454596265s" totalImagesPullingTime="5.03844664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:25.4538423 +0000 UTC m=+37.106116839" watchObservedRunningTime="2026-10-30 14:47:25.454596265 +0000 UTC m=+37.106870814"
Oct 30 14:47:25.515102 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.514077    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"]
Oct 30 14:47:25.534116 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.533389    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"]
Oct 30 14:47:25.550111 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.546976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:25.567862 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.567580    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"]
Oct 30 14:47:25.567862 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.567625    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"]
Oct 30 14:47:25.567862 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.567719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:25.572012 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.571982    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:25.572197 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.572133    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hg7s\""
Oct 30 14:47:25.572197 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.572193    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 14:47:25.658798 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.658758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflr4\" (UniqueName: \"kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4\") pod \"migrator-66df5d496b-xgd8m\" (UID: \"4f07f4f5-11c1-4474-8303-3f7cbd171a7d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:25.658985 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.658964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscxx\" (UniqueName: \"kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx\") pod \"migrator-66df5d496b-7qshm\" (UID: \"057216e1-88ce-443d-8033-657bfc04ad17\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:25.761011 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.760389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflr4\" (UniqueName: \"kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4\") pod \"migrator-66df5d496b-xgd8m\" (UID: \"4f07f4f5-11c1-4474-8303-3f7cbd171a7d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:25.761011 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.760492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscxx\" (UniqueName: \"kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx\") pod \"migrator-66df5d496b-7qshm\" (UID: \"057216e1-88ce-443d-8033-657bfc04ad17\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:25.769754 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.769723    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pg7ql"]
Oct 30 14:47:25.787480 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.787410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.788534 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.788511    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscxx\" (UniqueName: \"kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx\") pod \"migrator-66df5d496b-7qshm\" (UID: \"057216e1-88ce-443d-8033-657bfc04ad17\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:25.793596 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.793561    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pg7ql"]
Oct 30 14:47:25.794316 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.794289    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflr4\" (UniqueName: \"kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4\") pod \"migrator-66df5d496b-xgd8m\" (UID: \"4f07f4f5-11c1-4474-8303-3f7cbd171a7d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:25.802225 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.802198    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 14:47:25.802319 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.802283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 14:47:25.802319 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.802283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 14:47:25.802413 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.802390    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 14:47:25.802474 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.802456    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xtzxg\""
Oct 30 14:47:25.861974 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.861928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.862147 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.861985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.862147 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.862061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmgx\" (UniqueName: \"kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.871769 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.871741    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n86"]
Oct 30 14:47:25.875070 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.874877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:25.888449 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.888423    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:47:25.888636 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.888615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.902672 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.901914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:25.906111 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.906087    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\""
Oct 30 14:47:25.906334 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.906311    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 14:47:25.906411 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.906311    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 14:47:25.917887 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.917859    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n86"]
Oct 30 14:47:25.917997 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.917975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:25.917997 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.917987    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmgx\" (UniqueName: \"kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.965900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.966040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsrc\" (UniqueName: \"kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:25.969657 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.967142    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.973974 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.973953    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:25.993051 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:25.992958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmgx\" (UniqueName: \"kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:26.067001 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.066906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.067001 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.066985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.067192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsrc\" (UniqueName: \"kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.067192 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067346 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.067346 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067346 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067624 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.067595    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 14:47:26.067768 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.067680    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.567656611 +0000 UTC m=+38.219931135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : configmap "kube-rbac-proxy" not found
Oct 30 14:47:26.067768 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.067914 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.067814    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:26.067914 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.067877    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:26.567862256 +0000 UTC m=+38.220136800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:26.067999 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.067914    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.094232 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.094207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsrc\" (UniqueName: \"kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.131150 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.131110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:26.168937 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.168878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.169085 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.168975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.169085 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.169064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.169206 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.169115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.169532 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.169500    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.169706 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.169682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.170463 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.170438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.193984 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.193951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.215567 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.215462    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"]
Oct 30 14:47:26.223001 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:26.220694    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f07f4f5_11c1_4474_8303_3f7cbd171a7d.slice/crio-7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8 WatchSource:0}: Error finding container 7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8: Status 404 returned error can't find the container with id 7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8
Oct 30 14:47:26.265337 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.265301    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"]
Oct 30 14:47:26.270282 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:26.270255    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057216e1_88ce_443d_8033_657bfc04ad17.slice/crio-8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876 WatchSource:0}: Error finding container 8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876: Status 404 returned error can't find the container with id 8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876
Oct 30 14:47:26.288203 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.288173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:26.288590 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.288550    2579 scope.go:122] "RemoveContainer" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"
Oct 30 14:47:26.288798 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.288778    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7j8cn_openshift-console-operator(f3374e02-805e-4d08-97c9-2207b67e62f8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" podUID="f3374e02-805e-4d08-97c9-2207b67e62f8"
Oct 30 14:47:26.290127 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.290096    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm" event={"ID":"057216e1-88ce-443d-8033-657bfc04ad17","Type":"ContainerStarted","Data":"8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876"}
Oct 30 14:47:26.292528 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.292504    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz" event={"ID":"ba2f7212-f693-4d67-8b61-fc75f1ad0372","Type":"ContainerStarted","Data":"6fd4c94c75fc49879abcb2a8ea26ebe79ad25990cb2d1b75a9aacd89abff838d"}
Oct 30 14:47:26.294623 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.294600    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m" event={"ID":"4f07f4f5-11c1-4474-8303-3f7cbd171a7d","Type":"ContainerStarted","Data":"7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8"}
Oct 30 14:47:26.304234 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.304212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:26.472738 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.472702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:26.472917 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.472750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:26.472917 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.472787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:26.472917 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.472809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:26.473093 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.472974    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:26.473093 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.472994    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b65c48686-qftvr: secret "image-registry-tls" not found
Oct 30 14:47:26.473093 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.472975    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 14:47:26.473093 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473058    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls podName:8407bb01-c8e0-4624-bd80-c27459b6c96e nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.473038433 +0000 UTC m=+46.125312971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls") pod "image-registry-b65c48686-qftvr" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e") : secret "image-registry-tls" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.473100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.473091905 +0000 UTC m=+46.125366442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : configmap references non-existent config key: service-ca.crt
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473059    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-766565bc5f-b7l2r: secret "image-registry-tls" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473157    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473173    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls podName:cbee651d-49b7-4013-b903-da36898913cf nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.473160967 +0000 UTC m=+46.125435523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls") pod "image-registry-766565bc5f-b7l2r" (UID: "cbee651d-49b7-4013-b903-da36898913cf") : secret "image-registry-tls" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.472983    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.473183653 +0000 UTC m=+46.125458177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:26.474399 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.473210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs podName:9e34f75c-8c90-48eb-b5e6-4d363017a5f3 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.473201427 +0000 UTC m=+46.125475953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs") pod "router-default-58ddf469d5-lvlsc" (UID: "9e34f75c-8c90-48eb-b5e6-4d363017a5f3") : secret "router-metrics-certs-default" not found
Oct 30 14:47:26.480190 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.480162    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pg7ql"]
Oct 30 14:47:26.483344 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:26.483318    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd5b57e_2745_4bc1_8331_abac3f327a13.slice/crio-50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b WatchSource:0}: Error finding container 50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b: Status 404 returned error can't find the container with id 50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b
Oct 30 14:47:26.508225 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.508166    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz" podStartSLOduration=329.657631827 podStartE2EDuration="5m35.508148755s" totalImagesPullingTime="5.850516928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:26.504963869 +0000 UTC m=+38.157238409" watchObservedRunningTime="2026-10-30 14:47:26.508148755 +0000 UTC m=+38.160423302"
Oct 30 14:47:26.573636 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.573603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:26.573728 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.573690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.573773 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573743    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 14:47:26.573773 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.573760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.573785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573803    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert podName:bfad0d99-650b-4984-8afb-757f92bf7db9 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.573789373 +0000 UTC m=+46.226063898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m9l7n" (UID: "bfad0d99-650b-4984-8afb-757f92bf7db9") : secret "networking-console-plugin-cert" not found
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573820    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573890    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:27.57388254 +0000 UTC m=+39.226157068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:26.573916 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.573891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:26.574131 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573930    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.573919617 +0000 UTC m=+46.226194151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:26.574131 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573934    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 14:47:26.574131 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573969    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 14:47:26.574131 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.573977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:27.573965346 +0000 UTC m=+39.226239888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : configmap "kube-rbac-proxy" not found
Oct 30 14:47:26.574131 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:26.574006    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls podName:4ae91844-19fb-4008-89ce-5e427270dc96 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:34.573997252 +0000 UTC m=+46.226271781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v4588" (UID: "4ae91844-19fb-4008-89ce-5e427270dc96") : secret "samples-operator-tls" not found
Oct 30 14:47:26.719491 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:26.719347    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:47:26.723350 ip-10-0-138-143 kubenswrapper[2579]: W1030 14:47:26.723317    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad3a326_d346_4956_b8ef_d29ca51b43a1.slice/crio-512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2 WatchSource:0}: Error finding container 512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2: Status 404 returned error can't find the container with id 512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2
Oct 30 14:47:27.301984 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.301878    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql" event={"ID":"4dd5b57e-2745-4bc1-8331-abac3f327a13","Type":"ContainerStarted","Data":"65cc2187a09177c2ece82363c31ef5f7de8b95f4df390a969b637f54abecd475"}
Oct 30 14:47:27.301984 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.301948    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql" event={"ID":"4dd5b57e-2745-4bc1-8331-abac3f327a13","Type":"ContainerStarted","Data":"50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b"}
Oct 30 14:47:27.304344 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.304319    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" event={"ID":"bad3a326-d346-4956-b8ef-d29ca51b43a1","Type":"ContainerStarted","Data":"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"}
Oct 30 14:47:27.304456 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.304369    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" event={"ID":"bad3a326-d346-4956-b8ef-d29ca51b43a1","Type":"ContainerStarted","Data":"512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2"}
Oct 30 14:47:27.306645 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.306619    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxt6d" event={"ID":"b12b5bea-cee8-4db4-89a3-ddec80c8f82a","Type":"ContainerStarted","Data":"38a973622ec17e9e66c2e32aa7d6169c5c932461f3d8a1770e32133ba72cd5d1"}
Oct 30 14:47:27.306770 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.306706    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:27.397169 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.397107    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql" podStartSLOduration=2.397084482 podStartE2EDuration="2.397084482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:27.350097988 +0000 UTC m=+39.002372537" watchObservedRunningTime="2026-10-30 14:47:27.397084482 +0000 UTC m=+39.049359030"
Oct 30 14:47:27.398303 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.398133    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" podStartSLOduration=2.398118119 podStartE2EDuration="2.398118119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:27.396201712 +0000 UTC m=+39.048476282" watchObservedRunningTime="2026-10-30 14:47:27.398118119 +0000 UTC m=+39.050392665"
Oct 30 14:47:27.439374 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.439311    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zxt6d" podStartSLOduration=35.358891942 podStartE2EDuration="38.439293155s" totalImagesPullingTime="3.080401213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:27.437755188 +0000 UTC m=+39.090029735" watchObservedRunningTime="2026-10-30 14:47:27.439293155 +0000 UTC m=+39.091567702"
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.587525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:27.587616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:27.587751    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:27.587800    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:29.58778672 +0000 UTC m=+41.240061257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : configmap "kube-rbac-proxy" not found
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:27.588123    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:27.588238 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:27.588157    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:29.588146568 +0000 UTC m=+41.240421105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:28.311732 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:28.311695    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m" event={"ID":"4f07f4f5-11c1-4474-8303-3f7cbd171a7d","Type":"ContainerStarted","Data":"f275a1afc0920b666bc37c7e74b825ac72d83ec5b05e623260dc4fc73282e173"}
Oct 30 14:47:28.355043 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:28.354995    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m" podStartSLOduration=1.45401692 podStartE2EDuration="3.354979688s" totalImagesPullingTime="1.900962768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:28.353005546 +0000 UTC m=+40.005280096" watchObservedRunningTime="2026-10-30 14:47:28.354979688 +0000 UTC m=+40.007254285"
Oct 30 14:47:29.100075 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.100030    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:29.100260 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.100211    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:29.100559 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.100545    2579 scope.go:122] "RemoveContainer" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"
Oct 30 14:47:29.100799 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.100773    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7j8cn_openshift-console-operator(f3374e02-805e-4d08-97c9-2207b67e62f8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" podUID="f3374e02-805e-4d08-97c9-2207b67e62f8"
Oct 30 14:47:29.317810 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.317770    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm" event={"ID":"057216e1-88ce-443d-8033-657bfc04ad17","Type":"ContainerStarted","Data":"5f1160dfdcd966367bd2c96083fe1c63ce4306bbd510f4f2e4d2fc3f05a7281e"}
Oct 30 14:47:29.318741 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.318713    2579 scope.go:122] "RemoveContainer" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"
Oct 30 14:47:29.318920 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.318894    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7j8cn_openshift-console-operator(f3374e02-805e-4d08-97c9-2207b67e62f8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" podUID="f3374e02-805e-4d08-97c9-2207b67e62f8"
Oct 30 14:47:29.359188 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.359087    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm" podStartSLOduration=2.364064358 podStartE2EDuration="4.35907201s" totalImagesPullingTime="1.995007652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:29.357699457 +0000 UTC m=+41.009974002" watchObservedRunningTime="2026-10-30 14:47:29.35907201 +0000 UTC m=+41.011346598"
Oct 30 14:47:29.398287 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.398256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vnzrb_43d22383-d225-4613-baf3-30a0b288519e/dns-node-resolver/0.log"
Oct 30 14:47:29.609323 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.609243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:29.609480 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.609366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:29.609480 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.609417    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:29.609480 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.609456    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 14:47:29.609631 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.609499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.609479681 +0000 UTC m=+45.261754215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:29.609631 ip-10-0-138-143 kubenswrapper[2579]: E1030 14:47:29.609520    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.60950991 +0000 UTC m=+45.261784439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : configmap "kube-rbac-proxy" not found
Oct 30 14:47:29.991632 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:29.991609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96lz7_9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/node-ca/0.log"
Oct 30 14:47:30.323256 ip-10-0-138-143 kubenswrapper[2579]: I1030 14:47:30.323227    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pnhqm" event={"ID":"9a72ab8d-d509-46b8-809f-55a708439ffd","Type":"ContainerStarted","Data":"bacd55c575df53f0130b9965dd2c45cae982015b04a26556625be55cdefd7b45"}
Oct 30 14:47:30.491455 ip-10-0-138-143 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 14:47:30.502234 ip-10-0-138-143 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 14:47:30.502408 ip-10-0-138-143 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 14:47:30.502461 ip-10-0-138-143 systemd[1]: kubelet.service: Consumed 3.334s CPU time.
Oct 30 14:47:30.515335 ip-10-0-138-143 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 14:47:30.629956 ip-10-0-138-143 kubenswrapper[5064]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:47:30.629956 ip-10-0-138-143 kubenswrapper[5064]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 14:47:30.629956 ip-10-0-138-143 kubenswrapper[5064]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:47:30.629956 ip-10-0-138-143 kubenswrapper[5064]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632316    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632408    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632413    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632416    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632419    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.632409 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632422    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632425    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632428    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632431    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632435    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632441    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632449    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632453    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632458    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632461    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632464    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632467    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632469    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632472    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632474    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632477    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632480    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632482    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632485    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632488    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.632587 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632491    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632494    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632497    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632500    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632502    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632505    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632508    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632511    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632513    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632517    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632522    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632525    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632527    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632530    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632533    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632536    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632539    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632542    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632544    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.633121 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632547    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632549    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632553    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632556    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632558    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632561    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632563    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632568    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632571    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632574    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632576    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632579    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632581    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632584    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632587    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632589    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632592    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632595    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632598    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632600    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.633602 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632603    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632606    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632608    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632611    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632614    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632616    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632619    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632623    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632626    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632629    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632632    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632634    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632637    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632639    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632642    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632645    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632648    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632651    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632653    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.634147 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632656    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632659    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632662    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632665    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632667    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632670    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632672    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632675    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632677    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632680    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632683    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632686    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632688    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632691    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632693    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632696    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632699    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632701    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632704    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.634625 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632707    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632709    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632714    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.632719    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633546    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633553    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633556    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633559    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633561    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633564    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633567    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633570    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633573    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633576    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633579    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633581    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633584    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633587    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633590    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.635131 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633592    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633595    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633597    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633600    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633603    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633606    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633609    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633611    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633645    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633648    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633651    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633654    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633656    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633661    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633665    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633669    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633672    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633675    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633678    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633681    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.635686 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633684    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633686    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633689    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633691    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633694    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633697    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633700    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633703    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633706    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633709    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633711    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633715    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633718    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633722    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633726    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633729    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633732    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633735    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633738    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633742    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.636523 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633746    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633750    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633754    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633758    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633762    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633772    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633775    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633778    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633781    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633784    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633788    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633790    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633793    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633795    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633798    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633801    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633804    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633806    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633808    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633811    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.637214 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633813    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633816    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633819    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633822    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633824    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633840    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633843    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633846    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633849    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633851    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633854    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633857    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633859    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633862    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633865    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633868    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633872    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633877    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633879    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633882    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.637749 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633884    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633887    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633890    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633893    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633896    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633898    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633901    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633904    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633906    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633909    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.633911    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634420    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634426    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634429    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634432    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634435    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634438    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634441    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634443    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.638348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634446    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634449    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634451    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634454    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634457    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634459    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634462    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634464    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634467    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634470    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634475    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634478    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634481    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634483    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634487    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634489    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634492    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634495    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634498    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.638838 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634501    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634504    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634506    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634509    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634513    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634516    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634519    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634522    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634525    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634528    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634530    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634533    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634535    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634538    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634541    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634543    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634547    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634551    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634554    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.639348 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634558    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634561    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634564    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634566    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634570    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634573    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634575    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634578    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634581    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634583    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634586    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634589    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634592    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634595    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634598    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634600    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634603    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634606    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634608    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.639818 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634611    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634613    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634616    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634618    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634621    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634623    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634626    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634629    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634632    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634635    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634638    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634641    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634643    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634646    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634648    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634651    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634653    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634657    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634660    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634663    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.640299 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634665    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634668    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634671    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634673    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634676    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634679    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634681    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634684    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634687    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634690    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634692    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634696    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634698    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634701    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634703    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634706    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634709    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634711    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634713    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634716    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.640809 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.634718    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634816    5064 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634839    5064 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634850    5064 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634856    5064 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634861    5064 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634866    5064 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634871    5064 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634875    5064 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634881    5064 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634884    5064 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634888    5064 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634891    5064 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634894    5064 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634897    5064 flags.go:64] FLAG: --cgroup-root=""
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634900    5064 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634903    5064 flags.go:64] FLAG: --client-ca-file=""
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634907    5064 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634910    5064 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634940    5064 flags.go:64] FLAG: --cluster-domain=""
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634944    5064 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634948    5064 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634951    5064 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634955    5064 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 14:47:30.641326 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634963    5064 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634966    5064 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634969    5064 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634973    5064 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634976    5064 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634979    5064 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634982    5064 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634986    5064 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634989    5064 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634993    5064 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.634997    5064 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635000    5064 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635003    5064 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635006    5064 flags.go:64] FLAG: --enable-server="true"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635010    5064 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635015    5064 flags.go:64] FLAG: --event-burst="100"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635019    5064 flags.go:64] FLAG: --event-qps="50"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635023    5064 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635026    5064 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635030    5064 flags.go:64] FLAG: --eviction-hard=""
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635034    5064 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635037    5064 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635041    5064 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635044    5064 flags.go:64] FLAG: --eviction-soft=""
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635047    5064 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 14:47:30.641922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635050    5064 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635053    5064 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635057    5064 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635060    5064 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635063    5064 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635066    5064 flags.go:64] FLAG: --feature-gates=""
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635071    5064 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635074    5064 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635078    5064 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635081    5064 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635084    5064 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635087    5064 flags.go:64] FLAG: --help="false"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635090    5064 flags.go:64] FLAG: --hostname-override="ip-10-0-138-143.ec2.internal"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635093    5064 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635096    5064 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635100    5064 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635103    5064 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635107    5064 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635110    5064 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635113    5064 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635116    5064 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635120    5064 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635123    5064 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635127    5064 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 14:47:30.642537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635131    5064 flags.go:64] FLAG: --kube-reserved=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635135    5064 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635138    5064 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635141    5064 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635144    5064 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635147    5064 flags.go:64] FLAG: --lock-file=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635150    5064 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635153    5064 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635156    5064 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635162    5064 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635165    5064 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635168    5064 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635171    5064 flags.go:64] FLAG: --logging-format="text"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635174    5064 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635178    5064 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635181    5064 flags.go:64] FLAG: --manifest-url=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635184    5064 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635189    5064 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635192    5064 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635196    5064 flags.go:64] FLAG: --max-pods="110"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635199    5064 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635202    5064 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635205    5064 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635208    5064 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635211    5064 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 14:47:30.643183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635214    5064 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635217    5064 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635225    5064 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635228    5064 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635236    5064 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635240    5064 flags.go:64] FLAG: --pod-cidr=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635243    5064 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635247    5064 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635250    5064 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635253    5064 flags.go:64] FLAG: --port="10250"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635256    5064 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635259    5064 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-061c918d1d40b3899"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635262    5064 flags.go:64] FLAG: --qos-reserved=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635265    5064 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635268    5064 flags.go:64] FLAG: --register-node="true"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635271    5064 flags.go:64] FLAG: --register-with-taints=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635275    5064 flags.go:64] FLAG: --registry-burst="10"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635278    5064 flags.go:64] FLAG: --registry-qps="5"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635281    5064 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635284    5064 flags.go:64] FLAG: --reserved-memory=""
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635288    5064 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635292    5064 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635294    5064 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635298    5064 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635301    5064 flags.go:64] FLAG: --runonce="false"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635304    5064 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 14:47:30.643782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635307    5064 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635310    5064 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635313    5064 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635316    5064 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635319    5064 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635322    5064 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635325    5064 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635328    5064 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635331    5064 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635334    5064 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635338    5064 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635342    5064 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635345    5064 flags.go:64] FLAG: --system-cgroups=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635348    5064 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635353    5064 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635356    5064 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635359    5064 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635363    5064 flags.go:64] FLAG: --tls-min-version=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635366    5064 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635369    5064 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635372    5064 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635375    5064 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635378    5064 flags.go:64] FLAG: --v="2"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635383    5064 flags.go:64] FLAG: --version="false"
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635387    5064 flags.go:64] FLAG: --vmodule=""
Oct 30 14:47:30.644502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635391    5064 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.635395    5064 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635949    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635963    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635969    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635974    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635978    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635983    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635989    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.635996    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636001    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636006    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636010    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636015    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636025    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636029    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636034    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636042    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636046    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.645129 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636051    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636055    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636059    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636063    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636067    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636072    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636076    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636085    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636089    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636093    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636098    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636103    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636107    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636111    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636115    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636121    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636125    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636129    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636133    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636138    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.645606 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636147    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636151    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636155    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636159    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636164    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636168    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636172    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636177    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636181    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636185    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636191    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636196    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636200    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636210    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636214    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636219    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636224    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636228    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636232    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636236    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.646253 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636240    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636245    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636252    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636256    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636261    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636270    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636275    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636279    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636292    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636296    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636300    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636304    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636309    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636315    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636321    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636326    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636330    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636335    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636344    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.646790 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636348    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636353    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636358    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636366    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636370    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636456    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636466    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636469    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636472    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636475    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636478    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636481    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636484    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636487    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636489    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636494    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636497    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636500    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636505    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636508    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.647347 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636511    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636514    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636518    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636525    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636543    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636546    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636681    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636688    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636691    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636694    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.636735    5064 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636787    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636791    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636794    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636798    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636802    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.647862 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636805    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636807    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636810    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636813    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636816    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636818    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636821    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636824    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636848    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636853    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636857    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636860    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636866    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636869    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636872    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636874    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636877    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636880    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636883    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636885    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.648303 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636888    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636892    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636897    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636900    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636910    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636913    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636916    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636919    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636921    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636924    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636928    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636931    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636934    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636936    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636939    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636942    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636945    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636948    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636951    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.648924 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636953    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636956    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636959    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636961    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636964    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636969    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636972    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636975    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636977    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636980    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636983    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636985    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636989    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636991    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636994    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636996    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.636999    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637001    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637006    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.649515 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637008    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637011    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637014    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637016    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637019    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637023    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637027    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637030    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637033    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637036    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637039    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637042    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637045    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637048    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637051    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637053    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637056    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637059    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637063    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.650006 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637066    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637068    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637071    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637073    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637076    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637079    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637082    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637085    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637088    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637091    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637093    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637096    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637099    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637101    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637104    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637106    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637109    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637111    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637114    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.650475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637117    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637119    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637122    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637125    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.637129    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.640887    5064 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.640913    5064 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.640935    5064 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.640940    5064 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.640997    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641003    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641006    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641010    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641013    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641016    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641019    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641022    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641025    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641027    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.651072 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641031    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641034    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641037    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641040    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641043    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641046    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641048    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641051    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641055    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641059    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641063    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641066    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641070    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641073    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641076    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641078    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641081    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641084    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641087    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.651812 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641089    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641093    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641096    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641099    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641101    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641104    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641107    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641109    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641112    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641115    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641117    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641120    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641123    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641126    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641129    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641131    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641135    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641139    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.652439 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641142    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641144    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641147    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641150    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641152    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641156    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641158    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641161    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641164    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641166    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641169    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641172    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641174    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641177    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641180    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641182    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641185    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641188    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641191    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641193    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.652899 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641196    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641198    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641201    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641204    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641207    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641209    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641212    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641214    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641217    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641220    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641222    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641225    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641228    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641231    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641233    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641236    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641239    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641241    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641244    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.653399 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641247    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641250    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641253    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641255    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641258    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641261    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641263    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641266    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641269    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641271    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641275    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641278    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641280    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641283    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641285    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641288    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641290    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641293    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641296    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.653876 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641298    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.641326    5064 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641470    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641474    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641477    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641480    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641483    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641485    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641488    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641490    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641493    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641495    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641498    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641500    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641503    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641505    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.654328 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641508    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641510    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641513    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641515    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641518    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641520    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641523    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641525    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641529    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641531    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641534    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641537    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641541    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641544    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641547    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641550    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641552    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641555    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.654713 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641558    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641561    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641563    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641566    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641569    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641571    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641574    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641576    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641579    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641581    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641584    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641586    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641589    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641591    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641594    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641596    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641599    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641602    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641604    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641607    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.655282 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641609    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641612    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641614    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641618    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641620    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641623    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641626    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641628    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641631    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641634    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641636    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641639    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641646    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641648    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641651    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641653    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641656    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641659    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641662    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.655784 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641664    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641667    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641669    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641672    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641675    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641677    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641680    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641683    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641685    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641688    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641692    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641695    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641698    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641701    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641703    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641706    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641709    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641712    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641715    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.656374 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641717    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641720    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641723    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641725    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641728    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641732    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641736    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641738    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641741    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641744    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641746    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641749    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641752    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641755    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641757    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641760    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.656850 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.641783    5064 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641813    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641817    5064 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641821    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641824    5064 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641842    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641845    5064 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641851    5064 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641854    5064 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641858    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641861    5064 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641863    5064 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641866    5064 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641869    5064 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641871    5064 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641874    5064 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641877    5064 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641880    5064 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 14:47:30.657301 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641883    5064 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641885    5064 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641888    5064 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641892    5064 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641895    5064 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641899    5064 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641901    5064 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641904    5064 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641907    5064 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641909    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641912    5064 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641915    5064 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641917    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641920    5064 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641922    5064 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641925    5064 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641928    5064 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641931    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641934    5064 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641937    5064 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 14:47:30.657735 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641940    5064 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641942    5064 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641945    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641948    5064 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641951    5064 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641953    5064 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641956    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641958    5064 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641961    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641963    5064 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641966    5064 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641969    5064 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641972    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641975    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641978    5064 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641982    5064 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641984    5064 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641988    5064 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641990    5064 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641993    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 14:47:30.658417 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641996    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.641998    5064 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642001    5064 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642003    5064 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642007    5064 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642010    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642013    5064 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642015    5064 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642018    5064 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642021    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642023    5064 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642026    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642028    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642031    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642033    5064 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642035    5064 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642038    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642040    5064 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642043    5064 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 14:47:30.658980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642046    5064 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642048    5064 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642051    5064 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642053    5064 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642056    5064 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642059    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642062    5064 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642065    5064 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642069    5064 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642071    5064 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642074    5064 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642077    5064 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642079    5064 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642082    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642085    5064 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642087    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642090    5064 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642092    5064 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642095    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642097    5064 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 14:47:30.659447 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642100    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642102    5064 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642105    5064 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642107    5064 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642110    5064 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642112    5064 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642115    5064 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642117    5064 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642120    5064 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:30.642122    5064 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.642174    5064 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643235    5064 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643293    5064 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643466    5064 server.go:1014] "Starting client certificate rotation"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643609    5064 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643667    5064 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:41:49 +0000 UTC" deadline="2028-06-17 02:45:13.010092795 +0000 UTC"
Oct 30 14:47:30.660082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.643704    5064 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14291h57m42.366392095s"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.644056    5064 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.645166    5064 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.646266    5064 log.go:25] "Validated CRI v1 runtime API"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.647924    5064 log.go:25] "Validated CRI v1 image API"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.648679    5064 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 14:47:30.660473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.651264    5064 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa8d8dde-1826-4e04-85e9-4814f3b90aa0:/dev/nvme0n1p3 bc16899c-71fa-4562-a9c4-6b08ac89e97e:/dev/nvme0n1p4]
Oct 30 14:47:30.660658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.651290    5064 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03a570dbbc6da03788d6b7bf4235df2949868913df66848c77cc4b05e3c3e149/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03a570dbbc6da03788d6b7bf4235df2949868913df66848c77cc4b05e3c3e149/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99683769ba873075d3c607cfc699175d3a8f17ce0e13050f6462fafcb1b01161/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99683769ba873075d3c607cfc699175d3a8f17ce0e13050f6462fafcb1b01161/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b51d9b655bda7ef0a3ab2e1e5e2d8731a637884388124f31fb796babe4d9535f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b51d9b655bda7ef0a3ab2e1e5e2d8731a637884388124f31fb796babe4d9535f/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea24a14a7fda5d85849c60ef8989e4bfd07efd5a8e525889ced07262234c71a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea24a14a7fda5d85849c60ef8989e4bfd07efd5a8e525889ced07262234c71a3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6d0f2af294b8d5e5a62ebd7753a8a96202c9ef504329f189773253419c56b10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6d0f2af294b8d5e5a62ebd7753a8a96202c9ef504329f189773253419c56b10/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/057216e1-88ce-443d-8033-657bfc04ad17/volumes/kubernetes.io~projected/kube-api-access-sscxx:{mountpoint:/var/lib/kubelet/pods/057216e1-88ce-443d-8033-657bfc04ad17/volumes/kubernetes.io~projected/kube-api-access-sscxx major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~projected/kube-api-access-ln786:{mountpoint:/var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~projected/kube-api-access-ln786 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0badeea4-6775-4cf7-ad74-887fa9ee375a/volumes/kubernetes.io~projected/kube-api-access-kx2zt:{mountpoint:/var/lib/kubelet/pods/0badeea4-6775-4cf7-ad74-887fa9ee375a/volumes/kubernetes.io~projected/kube-api-access-kx2zt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~projected/kube-api-access-9zttn:{mountpoint:/var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~projected/kube-api-access-9zttn major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26527e26-b484-446f-b704-ee332fea2f2e/volumes/kubernetes.io~projected/kube-api-access-lwp8v:{mountpoint:/var/lib/kubelet/pods/26527e26-b484-446f-b704-ee332fea2f2e/volumes/kubernetes.io~projected/kube-api-access-lwp8v major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30c328c3-1bca-4576-9fd0-c669059e423e/volumes/kubernetes.io~projected/kube-api-access-6vprm:{mountpoint:/var/lib/kubelet/pods/30c328c3-1bca-4576-9fd0-c669059e423e/volumes/kubernetes.io~projected/kube-api-access-6vprm major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43d22383-d225-4613-baf3-30a0b288519e/volumes/kubernetes.io~projected/kube-api-access-qt98x:{mountpoint:/var/lib/kubelet/pods/43d22383-d225-4613-baf3-30a0b288519e/volumes/kubernetes.io~projected/kube-api-access-qt98x major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ae91844-19fb-4008-89ce-5e427270dc96/volumes/kubernetes.io~projected/kube-api-access-rvzwc:{mountpoint:/var/lib/kubelet/pods/4ae91844-19fb-4008-89ce-5e427270dc96/volumes/kubernetes.io~projected/kube-api-access-rvzwc major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~projected/kube-api-access-wlmgx:{mountpoint:/var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~projected/kube-api-access-wlmgx major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~secret/signing-key major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f07f4f5-11c1-4474-8303-3f7cbd171a7d/volumes/kubernetes.io~projected/kube-api-access-kflr4:{mountpoint:/var/lib/kubelet/pods/4f07f4f5-11c1-4474-8303-3f7cbd171a7d/volumes/kubernetes.io~projected/kube-api-access-kflr4 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d35a9c-d95f-4eba-a095-1f95b931f16a/volumes/kubernetes.io~projected/kube-api-access-jvm2z:{mountpoint:/var/lib/kubelet/pods/71d35a9c-d95f-4eba-a095-1f95b931f16a/volumes/kubernetes.io~projected/kube-api-access-jvm2z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~projected/kube-api-access-v7wft:{mountpoint:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~projected/kube-api-access-v7wft major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82e0ad12-29e4-4892-841b-801edb30b567/volumes/kubernetes.io~projected/kube-api-access-nbr89:{mountpoint:/var/lib/kubelet/pods/82e0ad12-29e4-4892-841b-801edb30b567/volumes/kubernetes.io~projected/kube-api-access-nbr89 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/kube-api-access-sv4fm:{mountpoint:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/kube-api-access-sv4fm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9526eb6d-d032-49ab-8a4e-901e901681a7/volumes/kubernetes.io~projected/kube-api-access-wp9nn:{mountpoint:/var/lib/kubelet/pods/9526eb6d-d032-49ab-8a4e-901e901681a7/volumes/kubernetes.io~projected/kube-api-access-wp9nn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a72ab8d-d509-46b8-809f-55a708439ffd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9a72ab8d-d509-46b8-809f-55a708439ffd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/volumes/kubernetes.io~projected/kube-api-access-q2gzn:{mountpoint:/var/lib/kubelet/pods/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/volumes/kubernetes.io~projected/kube-api-access-q2gzn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~projected/kube-api-access-bcpr6:{mountpoint:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~projected/kube-api-access-bcpr6 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b12b5bea-cee8-4db4-89a3-ddec80c8f82a/volumes/kubernetes.io~projected/kube-api-access-2kgzb:{mountpoint:/var/lib/kubelet/pods/b12b5bea-cee8-4db4-89a3-ddec80c8f82a/volumes/kubernetes.io~projected/kube-api-access-2kgzb major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~projected/kube-api-access-frz62:{mountpoint:/var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~projected/kube-api-access-frz62 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~projected/kube-api-access-bwr2p:{mountpoint:/var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~projected/kube-api-access-bwr2p major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~secret/cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2f7212-f693-4d67-8b61-fc75f1ad0372/volumes/kubernetes.io~projected/kube-api-access-hh5k9:{mountpoint:/var/lib/kubelet/pods/ba2f7212-f693-4d67-8b61-fc75f1ad0372/volumes/kubernetes.io~projected/kube-api-access-hh5k9 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bad3a326-d346-4956-b8ef-d29ca51b43a1/volumes/kubernetes.io~projected/kube-api-access-mvbhg:{mountpoint:/var/lib/kubelet/pods/bad3a326-d346-4956-b8ef-d29ca51b43a1/volumes/kubernetes.io~projected/kube-api-access-mvbhg major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bef2ba8e-37ad-41c2-a322-b481ba3be1e1/volumes/kubernetes.io~projected/kube-api-access-lhh59:{mountpoint:/var/lib/kubelet/pods/bef2ba8e-37ad-41c2-a322-b481ba3be1e1/volumes/kubernetes.io~projected/kube-api-access-lhh59 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/kube-api-access-rdw76:{mountpoint:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/kube-api-access-rdw76 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d375dcba-a3e9-40ec-9e41-fb7e692430f4/volumes/kubernetes.io~projected/kube-api-access-kwpt7:{mountpoint:/var/lib/kubelet/pods/d375dcba-a3e9-40ec-9e41-fb7e692430f4/volumes/kubernetes.io~projected/kube-api-access-kwpt7 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dee6e3a3-595c-4f16-a095-4e79aae97230/volumes/kubernetes.io~projected/kube-api-access-tzqqn:{mountpoint:/var/lib/kubelet/pods/dee6e3a3-595c-4f16-a095-4e79aae97230/volumes/kubernetes.io~projected/kube-api-access-tzqqn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~projected/kube-api-access-bzdx7:{mountpoint:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~projected/kube-api-access-bzdx7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebdd075b-3f66-4f77-ab73-e8f080e78f37/volumes/kubernetes.io~projected/kube-api-access-pgxdr:{mountpoint:/var/lib/kubelet/pods/ebdd075b-3f66-4f77-ab73-e8f080e78f37/volumes/kubernetes.io~projected/kube-api-access-pgxdr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8/volumes/kubernetes.io~projected/kube-api-access-bmksh:{mountpoint:/var/lib/kubelet/pods/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8/volumes/kubernetes.io~projected/kube-api-access-bmksh major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~projected/kube-api-access-f8vpn:{mountpoint:/var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~projected/kube-api-access-f8vpn major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f43cc708-9756-48e8-a788-a79a9a016e9a/volumes/kubernetes.io~projected/kube-api-access-zfsrc:{mountpoint:/var/lib/kubelet/pods/f43cc708-9756-48e8-a788-a79a9a016e9a/volumes/kubernetes.io~projected/kube-api-access-zfsrc major:0 minor:340 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7ce359ae2419b14827d2dd27607265488012ac313959b7708c0e0fa0bf9a6e6b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/186f23859a3f14266093bc4c219167e6c5c5fb734230ff2a0ef925aa3dfc709e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/71a0de8d725340b055f633caec93c6b6fb8aa8727ea2ab65959e65da3ae560bc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/518fe6e72db2bcfe5c65453934ff8fbef5ee50dbd32c29d8fe9216a857f69320/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9e8444b63f4d9bd723cc70f80d176b2a28286a98f8cbdcd9072913d469e39b7d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0a6d1b6b518ac9ece0af69544ef103960e7d8eb60c5bf67e76aba5a40ca80ce1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4a325ebc6b0e25d94c2f7a6d787d41f38ddb4fbee8c0730a18ffebad4db627ae/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/20d0ecbcf4b6f5400d6a7ad0098156f9539ce6ae11ad81f47562463f30116872/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f60ed0a65c31b598cffc29e81af6ce976e038fb7d55e449480c0e3bf0c219a7e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b1727bbbe47c5f1b5135f29c8676465a5827c318bb8c05711e2c29d13f013275/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/210d69ed12044182c059139ab16026e9f989ad6c717527c8ee372dd525c09b62/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f507dea4acd7482788902f2032c2fec3f40980a01515b0ae04a8d9c9986563bf/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8a81b3c7e8867e9bf3c8697b2eb855217f3ae85883690ab6767a3692f6ecc7cf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/40f962a0d0675cc5e5de0c1c71971aa310806e2fc10de184fac09603457d6847/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5e1ca26330e5af2f08b6c6a4739b4072b20b5447fb38fc761cb500403cc6dcaf/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/889e4da166ae467ec7d0ab2a1e4aa21a1038a4a8ed2dd4f594aaad0a4f4fe94e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dcfb32258b3e14cf75d3449871588b270ef2b412eeb4e64577e3d5a6f3b2d16f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2acab70c2c442c4478138446c4c44f5d2b3bcf0d1c791ef9d45c7c233add1e9b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dcfeaf21f94186f6681cf07d1b836ceda05ef55f7618d228a2f2f91fc2c575ce/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2b0bb80dedbce64ed9572cfc13a1c39f79542a2eaa63eb39a4a302a67bd28603/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fb93d801a22565d31fd71ac2c9655b63d26a9e1eb58aa996a1af955f366ca5b3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/377b7f7bad2c678c831f646c2c14b4d13e1d2474ed20bfe9a02558a83f49e780/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5d7c0d46398b2937aba0a5d3d814a69483f7e24bf91f02a869c4c28a1a200d29/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a36196d5fe9dc241c30c934fe9dc62b17e518f3ffad0f850886cafb2bd14ce41/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/41520a707b06de368202e359bb01df32ac79ac496949b432453f2619e378c448/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f8c138267593a27306769f097cf68e155dd99afebd067f08ba2feb1a748a29ec/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/497670c306280b2b862c9923b118e21a8f00d042f39295c6c9ea6e2eef09dbb1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/3efd8339e0793e3b4b27114dc5495218275454a87c62824e9feaec66a5f94c38/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5ab927881ae3d2011a9eeb9f02ec18448a3b2b5bdbfe281a4c17f7601c40085e/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d6b54b0aa133b63cb287dcd0dcca580989bc184519dc59952c091b0f16d6b4f8/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/1ad8bd58317fd0561cb0d9700c0465ba1b136d0a6b43a1a6a0293cb6318cb69b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/76309f4b55001634b574a1f88c9a9797371950dd3db080715ab17de76fe222e0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/0a96763997c91c2cc75ccc3791d10ed4d5b5f3bb92e31ac45596e751eef04d79/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f425bc19518470f0ed0a3151041408dd8cc108295e99095d3cc39062b762b135/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/4cdd0bbba7b29a05adf73e3676cf98e59ceae81d7cf649970fb00eb61ce4129b/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9d46d64dd1ee0d2b9d182245d796ee513160c5311389a4dc02e25815831b5952/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8819318f75156fbbb42026405d72d22d09923591b5bfe37421c6641078bd38a0/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/ddaf48266f68c9a97dcd2defff3177fb7cad9abbf89cc600027bfe58d91fbca7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6a1ac6c2955e3361d52d984e4bae48fd8e1f8cc79881ae39c61eb62260a81b05/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/abac2791feda23980bf5bd2e547cc348359abe5c7866a3518ab4bdaf03367fc0/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/342650c8a661117bf7599c3adf36e68e8eb28ae4841ca129868861cda2c324ab/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/61f674c5f0342a99733fab15d7da37cce9da20147e18f42bf03fed2bb07e8821/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/ae24702df137bd851833fe895c666be81742b2f41582f73e7f709cf6287f9d7d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/cc96edf1d2d61a96179c7e650b9821505d9e19606a320e856cccc1f22182d262/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/17bc69c5e656dabee29ca61a9422710cb2bd8cc7cf5a3365132a11957f95de37/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a4845b9ee63bbdced7d62d1a884c7952c210eb526a460a79f6423e35b0ef199f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/793478679d3f92e34bd82cb26ee5d7e3791b2218ebc1b495e3f3522252944c85/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ba1c398f6787af57cc9137391388818316f2dc5055b10d31b369f6cdb90f4305/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/6028dee95fb9de10584accf4d296437324153eb02c56702be21a5bb79cd00c8c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/1e1010b9c8effafc5572dac56df82aeb877e48705154ce4ab61ddbe704215ee6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5125d58d727c5f88e3f792c07a843c7eba560b0b71d519dfb488e7fc9c815e6a/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/25c61e16f57fd48f04b64f59faf21c299c351f01c8b39df662ffb22c190056a5/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/d6907da5d911fd3e560a6dd76dcedb10f0379f8922bbc067caa470f9bbce0b44/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b6076b7ed5e4f381b9692cd568c00df272e8d3899917b2779c6d51441eeb19b6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/115d3e69920cdac128e5b80ac5b5d1ad10ff5e7388682e9813c229f77c764989/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/578ee53058bda4215869bc2d1ecf213d92fc7f4a80b9f22b08e76a94ebd2eda8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e39229d98af89b711ada6652baae973095e002025d5c422d122f63adce036664/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/563b2377d2ce3bd584b2b0c2a95ff75df16d1c58dfc29becda5208fa6c622abf/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d48f4d5cfabf0f61db22c991ffc1b13328a9f3b639b25e4ce4bc7c653b80f41e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/82c997422f1092a22d7561f2c23bcca3f98fc7ed94d102b95fff06a995d96ac4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/3215f51bd187f3116622563dc054ca8fd69ef1158151701a708aa5bdf2024b23/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ce5532b4c499e0c7fa1e71622d317e4df1281b0c637d88182ea39c88cbe1d0e1/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 14:47:30.661385 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661006    5064 manager.go:217] Machine: {Timestamp:2026-10-30 14:47:30.660516286 +0000 UTC m=+0.102406997 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099577 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2220afab71d0e7dee398c485fd5b2b SystemUUID:ec2220af-ab71-d0e7-dee3-98c485fd5b2b BootID:8be41e49-17e9-4e9a-8f14-7bcc2aa83678 Filesystems:[{Device:/var/lib/kubelet/pods/ba2f7212-f693-4d67-8b61-fc75f1ad0372/volumes/kubernetes.io~projected/kube-api-access-hh5k9 DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3be014ff1967c090ba854a0d0826ce7cf783f192e28487a90508e59592c68db1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1fb287caa0b36eef9d6a43cfb111f229b625ef97f8428bde270456dfde39efc/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6d0f2af294b8d5e5a62ebd7753a8a96202c9ef504329f189773253419c56b10/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b51d9b655bda7ef0a3ab2e1e5e2d8731a637884388124f31fb796babe4d9535f/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/566beee457d5113142b3a39c1333d470160bd0e8007c423686dbc12943b385ac/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~projected/kube-api-access-bzdx7 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dba7d64a1bbb5110bbd0a4dfb160d5357d9615c8ed316fbb23311b548b812061/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/kube-api-access-rdw76 DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30c328c3-1bca-4576-9fd0-c669059e423e/volumes/kubernetes.io~projected/kube-api-access-6vprm DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/26527e26-b484-446f-b704-ee332fea2f2e/volumes/kubernetes.io~projected/kube-api-access-lwp8v DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bef2ba8e-37ad-41c2-a322-b481ba3be1e1/volumes/kubernetes.io~projected/kube-api-access-lhh59 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6517510a32a280e0e6e1fe4a47376b8687a2990d4082a085f1bbcd9b24704b4/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cd264438cf5c0c115219c0d7df335ede7b266e875f368369d85daa57a533876/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82e0ad12-29e4-4892-841b-801edb30b567/volumes/kubernetes.io~projected/kube-api-access-nbr89 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~projected/kube-api-access-bcpr6 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a72ab8d-d509-46b8-809f-55a708439ffd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bad3a326-d346-4956-b8ef-d29ca51b43a1/volumes/kubernetes.io~projected/kube-api-access-mvbhg DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50f16a6e3995a3ae5087f7207f19e9a0e6f00620255ad3ba7a208d369909979b/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea952af6-f7f1-43d6-82b4-8137b837a4e0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f9e692-eeb3-4000-80dc-0eebeb13f1db/volumes/kubernetes.io~projected/kube-api-access-bwr2p DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03a570dbbc6da03788d6b7bf4235df2949868913df66848c77cc4b05e3c3e149/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3ab7030ea97de45c32d852f212367bec8f20fbdb844c374cc8f6b7f346db1a3/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b12b5bea-cee8-4db4-89a3-ddec80c8f82a/volumes/kubernetes.io~projected/kube-api-access-2kgzb DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f43cc708-9756-48e8-a788-a79a9a016e9a/volumes/kubernetes.io~projected/kube-api-access-zfsrc DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43d22383-d225-4613-baf3-30a0b288519e/volumes/kubernetes.io~projected/kube-api-access-qt98x DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adb4789a51bfa8fbd21b9a55c2125cd17450da004a97570714611f42e05ca977/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9473e91dd32b03eb03c5a24ca9f64f7038fb9065d7d422250f3bb55fa42da0b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebdd075b-3f66-4f77-ab73-e8f080e78f37/volumes/kubernetes.io~projected/kube-api-access-pgxdr DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e3ddf7d46226f6dd0030ec59679f3b48d3de2a6bc804bb17abef657d7d05858/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d517e6932ab655b503e439cf5db8f84fd434335eab32337fbc10beee889e1779/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~projected/kube-api-access-9zttn DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b364c387bb78586b789132dba9d8d670a2111eb87dea65a15e739d70cf5e3cf2/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f07f4f5-11c1-4474-8303-3f7cbd171a7d/volumes/kubernetes.io~projected/kube-api-access-kflr4 DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~projected/kube-api-access-wlmgx DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae655be98dd3bba552699052c1761de915fde382aa4a7681e92af131cf4e1ef/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e34f75c-8c90-48eb-b5e6-4d363017a5f3/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f541756332567e50d5d05abe2c88cf4c5fa4bc95b539c70b92407592a3305a8/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0badeea4-6775-4cf7-ad74-887fa9ee375a/volumes/kubernetes.io~projected/kube-api-access-kx2zt DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~projected/kube-api-access-frz62 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d375dcba-a3e9-40ec-9e41-fb7e692430f4/volumes/kubernetes.io~projected/kube-api-access-kwpt7 DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d35f341c90a1a078eac84824a9bfb2b79f6542a9b0330c4de8ab92a59e7306da/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd5b57e-2745-4bc1-8331-abac3f327a13/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/107da892-9033-4172-b313-2431ef06d16c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ae91844-19fb-4008-89ce-5e427270dc96/volumes/kubernetes.io~projected/kube-api-access-rvzwc DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/057216e1-88ce-443d-8033-657bfc04ad17/volumes/kubernetes.io~projected/kube-api-access-sscxx DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dee6e3a3-595c-4f16-a095-4e79aae97230/volumes/kubernetes.io~projected/kube-api-access-tzqqn DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ec3816-9582-4e08-b9d0-9d03174f5628/volumes/kubernetes.io~projected/kube-api-access-ln786 DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71d35a9c-d95f-4eba-a095-1f95b931f16a/volumes/kubernetes.io~projected/kube-api-access-jvm2z DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12fb0d2f2e113350e32dd9f55aa7a6bd79f13d82b26f2a8a6072ffbdd691d486/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/volumes/kubernetes.io~projected/kube-api-access-q2gzn DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b767b7e7-8048-4086-aa1b-5ec54e3d0568/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9526eb6d-d032-49ab-8a4e-901e901681a7/volumes/kubernetes.io~projected/kube-api-access-wp9nn DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~projected/kube-api-access-sv4fm DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea24a14a7fda5d85849c60ef8989e4bfd07efd5a8e525889ced07262234c71a3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8/volumes/kubernetes.io~projected/kube-api-access-bmksh DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d960551a0b6a68e58f61e79494bf205329b1765273100c852ae212228be12eda/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79986431-390d-4313-af30-b2ff62e0ce68/volumes/kubernetes.io~projected/kube-api-access-v7wft DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3374e02-805e-4d08-97c9-2207b67e62f8/volumes/kubernetes.io~projected/kube-api-access-f8vpn DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99683769ba873075d3c607cfc699175d3a8f17ce0e13050f6462fafcb1b01161/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03a570dbbc6da03 MacAddress:82:f4:dd:b3:94:02 Speed:10000 Mtu:8901} {Name:12fb0d2f2e11335 MacAddress:ae:06:7d:7e:62:0f Speed:10000 Mtu:8901} {Name:3be014ff1967c09 MacAddress:ca:10:18:f1:3c:da Speed:10000 Mtu:8901} {Name:50f16a6e3995a3a MacAddress:4e:c1:39:00:a4:44 Speed:10000 Mtu:8901} {Name:512b71d6ec5b87b MacAddress:26:7e:be:69:28:a8 Speed:10000 Mtu:8901} {Name:5e3ddf7d46226f6 MacAddress:de:c3:7a:cf:80:bf Speed:10000 Mtu:8901} {Name:7f541756332567e MacAddress:b6:6e:c8:10:cb:3a Speed:10000 Mtu:8901} {Name:8cd264438cf5c0c MacAddress:ca:cc:bb:63:31:31 Speed:10000 Mtu:8901} {Name:99683769ba87307 MacAddress:72:a9:f6:29:1f:8e Speed:10000 Mtu:8901} {Name:b364c387bb78586 MacAddress:52:6c:af:62:b9:b0 Speed:10000 Mtu:8901} {Name:b51d9b655bda7ef MacAddress:46:2e:cf:c8:12:a8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:09:db:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:48:ff:8e:e2:0f Speed:0 Mtu:8901} {Name:d35f341c90a1a07 MacAddress:8e:5f:eb:fd:71:00 Speed:10000 Mtu:8901} {Name:d6517510a32a280 MacAddress:d6:9b:76:a0:d4:66 Speed:10000 Mtu:8901} {Name:d960551a0b6a68e MacAddress:56:b0:e2:ea:0a:66 Speed:10000 Mtu:8901} {Name:e3ab7030ea97de4 MacAddress:32:6a:51:04:24:6c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:09:db:7b Speed:0 Mtu:9001} {Name:f6d0f2af294b8d5 MacAddress:f6:1d:d9:b1:4b:17 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a6:e6:02:df:15:aa Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:cd:a4:11:6a:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 14:47:30.661577 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661390    5064 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 14:47:30.661577 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661441    5064 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 14:47:30.661791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661760    5064 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 14:47:30.661982 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661792    5064 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 14:47:30.662031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661986    5064 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 14:47:30.662031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.661994    5064 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 14:47:30.662031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662008    5064 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 14:47:30.662031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662022    5064 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 14:47:30.662132 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662042    5064 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 14:47:30.662236 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662227    5064 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 14:47:30.662347 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662338    5064 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 14:47:30.662381 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662375    5064 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 14:47:30.662408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662386    5064 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 14:47:30.662408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662398    5064 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 14:47:30.662408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662408    5064 kubelet.go:405] "Adding apiserver pod source"
Oct 30 14:47:30.662490 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.662415    5064 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 14:47:30.663491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.663475    5064 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 14:47:30.663491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.663487    5064 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 14:47:30.664103 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.664076    5064 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 14:47:30.666947 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.666930    5064 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 14:47:30.666947 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.666945    5064 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 14:47:30.667084 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667055    5064 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 14:47:30.667295 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667280    5064 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 14:47:30.667359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667320    5064 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 14:47:30.667448 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667437    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 14:47:30.667502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667457    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 14:47:30.667502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667467    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 14:47:30.667502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667477    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 14:47:30.667502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667486    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 14:47:30.667502 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667496    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667505    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667514    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667527    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667536    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667550    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667565    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667606    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 14:47:30.667739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.667617    5064 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 14:47:30.671345 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.671327    5064 server.go:1263] "Started kubelet"
Oct 30 14:47:30.671429 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.671390    5064 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 14:47:30.671970 ip-10-0-138-143 systemd[1]: Started Kubernetes Kubelet.
Oct 30 14:47:30.677016 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.674116    5064 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 14:47:30.677016 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.674685    5064 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 14:47:30.677016 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.675151    5064 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 14:47:30.677325 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.677288    5064 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 14:47:30.679009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.678273    5064 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 14:47:30.679009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.678489    5064 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 14:47:30.679009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.678523    5064 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 14:47:30.679009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.678581    5064 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 14:47:30.679009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.678287    5064 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 14:47:30.679278 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.679024    5064 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:41:51 +0000 UTC" deadline="2028-05-23 01:53:55.744003294 +0000 UTC"
Oct 30 14:47:30.679278 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.679079    5064 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13691h6m25.064931206s"
Oct 30 14:47:30.679278 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:30.679122    5064 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-143.ec2.internal\" not found"
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.680591    5064 factory.go:55] Registering systemd factory
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.680611    5064 factory.go:223] Registration of the systemd container factory successfully
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682847    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ae91844-19fb-4008-89ce-5e427270dc96" volumeName="kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682897    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682915    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682930    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682943    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9526eb6d-d032-49ab-8a4e-901e901681a7" volumeName="kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682956    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e34f75c-8c90-48eb-b5e6-4d363017a5f3" volumeName="kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682967    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f9e692-eeb3-4000-80dc-0eebeb13f1db" volumeName="kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682979    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.682999    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683011    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4" volumeName="kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683024    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683035    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683046    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8" volumeName="kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683057    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3374e02-805e-4d08-97c9-2207b67e62f8" volumeName="kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683071    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3374e02-805e-4d08-97c9-2207b67e62f8" volumeName="kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683082    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3374e02-805e-4d08-97c9-2207b67e62f8" volumeName="kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683104    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad3a326-d346-4956-b8ef-d29ca51b43a1" volumeName="kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683115    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79986431-390d-4313-af30-b2ff62e0ce68" volumeName="kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683126    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="057216e1-88ce-443d-8033-657bfc04ad17" volumeName="kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683138    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ec3816-9582-4e08-b9d0-9d03174f5628" volumeName="kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683149    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ec3816-9582-4e08-b9d0-9d03174f5628" volumeName="kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683160    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683177    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ae91844-19fb-4008-89ce-5e427270dc96" volumeName="kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683188    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683200    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d35a9c-d95f-4eba-a095-1f95b931f16a" volumeName="kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683213    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d375dcba-a3e9-40ec-9e41-fb7e692430f4" volumeName="kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683225    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bef2ba8e-37ad-41c2-a322-b481ba3be1e1" volumeName="kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683235    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683253    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f43cc708-9756-48e8-a788-a79a9a016e9a" volumeName="kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683265    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9526eb6d-d032-49ab-8a4e-901e901681a7" volumeName="kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683277    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfad0d99-650b-4984-8afb-757f92bf7db9" volumeName="kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683304    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea952af6-f7f1-43d6-82b4-8137b837a4e0" volumeName="kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683316    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd5b57e-2745-4bc1-8331-abac3f327a13" volumeName="kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683328    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d22383-d225-4613-baf3-30a0b288519e" volumeName="kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683341    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad3a326-d346-4956-b8ef-d29ca51b43a1" volumeName="kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683352    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3" volumeName="kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683364    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683376    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d375dcba-a3e9-40ec-9e41-fb7e692430f4" volumeName="kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683388    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee6e3a3-595c-4f16-a095-4e79aae97230" volumeName="kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683401    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683415    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683427    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d35a9c-d95f-4eba-a095-1f95b931f16a" volumeName="kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683438    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9526eb6d-d032-49ab-8a4e-901e901681a7" volumeName="kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683450    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a72ab8d-d509-46b8-809f-55a708439ffd" volumeName="kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683466    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683476    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683488    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbee651d-49b7-4013-b903-da36898913cf" volumeName="kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683500    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd5b57e-2745-4bc1-8331-abac3f327a13" volumeName="kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683512    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79986431-390d-4313-af30-b2ff62e0ce68" volumeName="kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683524    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e34f75c-8c90-48eb-b5e6-4d363017a5f3" volumeName="kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683535    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad3a326-d346-4956-b8ef-d29ca51b43a1" volumeName="kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683546    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4" volumeName="kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683559    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b767b7e7-8048-4086-aa1b-5ec54e3d0568" volumeName="kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683570    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f07f4f5-11c1-4474-8303-3f7cbd171a7d" volumeName="kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683586    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683598    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee6e3a3-595c-4f16-a095-4e79aae97230" volumeName="kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683619    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebdd075b-3f66-4f77-ab73-e8f080e78f37" volumeName="kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683630    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f43cc708-9756-48e8-a788-a79a9a016e9a" volumeName="kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683644    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea952af6-f7f1-43d6-82b4-8137b837a4e0" volumeName="kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683657    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0badeea4-6775-4cf7-ad74-887fa9ee375a" volumeName="kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683669    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683682    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30c328c3-1bca-4576-9fd0-c669059e423e" volumeName="kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683694    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd5b57e-2745-4bc1-8331-abac3f327a13" volumeName="kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683706    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b767b7e7-8048-4086-aa1b-5ec54e3d0568" volumeName="kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683723    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f9e692-eeb3-4000-80dc-0eebeb13f1db" volumeName="kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683736    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad3a326-d346-4956-b8ef-d29ca51b43a1" volumeName="kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683748    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bef2ba8e-37ad-41c2-a322-b481ba3be1e1" volumeName="kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683760    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82e0ad12-29e4-4892-841b-801edb30b567" volumeName="kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683772    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8407bb01-c8e0-4624-bd80-c27459b6c96e" volumeName="kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683784    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b12b5bea-cee8-4db4-89a3-ddec80c8f82a" volumeName="kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683796    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bef2ba8e-37ad-41c2-a322-b481ba3be1e1" volumeName="kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683808    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3" volumeName="kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683819    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee6e3a3-595c-4f16-a095-4e79aae97230" volumeName="kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683856    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea952af6-f7f1-43d6-82b4-8137b837a4e0" volumeName="kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683868    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea952af6-f7f1-43d6-82b4-8137b837a4e0" volumeName="kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683879    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79986431-390d-4313-af30-b2ff62e0ce68" volumeName="kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683892    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d22383-d225-4613-baf3-30a0b288519e" volumeName="kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683907    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ec3816-9582-4e08-b9d0-9d03174f5628" volumeName="kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683927    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9526eb6d-d032-49ab-8a4e-901e901681a7" volumeName="kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683939    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e34f75c-8c90-48eb-b5e6-4d363017a5f3" volumeName="kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683950    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b767b7e7-8048-4086-aa1b-5ec54e3d0568" volumeName="kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683962    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea952af6-f7f1-43d6-82b4-8137b837a4e0" volumeName="kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683974    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107da892-9033-4172-b313-2431ef06d16c" volumeName="kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683986    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26527e26-b484-446f-b704-ee332fea2f2e" volumeName="kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.683998    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2f7212-f693-4d67-8b61-fc75f1ad0372" volumeName="kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.684011    5064 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3374e02-805e-4d08-97c9-2207b67e62f8" volumeName="kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn" seLinuxMountContext=""
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.684028    5064 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 14:47:30.685368 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.684037    5064 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687189    5064 factory.go:153] Registering CRI-O factory
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687205    5064 factory.go:223] Registration of the crio container factory successfully
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687260    5064 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687280    5064 factory.go:103] Registering Raw factory
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687294    5064 manager.go:1199] Started watching for new ooms in manager
Oct 30 14:47:30.689527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.687622    5064 manager.go:319] Starting recovery of all containers
Oct 30 14:47:30.700304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.700277    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:30.700445 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.700418    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:30.712639 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.712582    5064 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 14:47:30.714622 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.714476    5064 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 14:47:30.714622 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.714499    5064 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 14:47:30.714622 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.714520    5064 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 14:47:30.714622 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:30.714565    5064 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 14:47:30.739305 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739271    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="dfda339a6dc54c7a9eddad94a8c367c1d67550b5b8f1ea3da49198facc9fe572" exitCode=0
Oct 30 14:47:30.739305 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739300    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="7aedb05499d9b2fd4eb81900703b20e286d7f8d2fbc5aaf43c47b8e1f049c817" exitCode=0
Oct 30 14:47:30.739305 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739308    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="bbd7b01f9bcaf8230157dc6a0b3951df5ff92c60cfd0d94b7ff4a0d941c55b70" exitCode=0
Oct 30 14:47:30.739305 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739315    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="b7164e6e047b41f69081a04f24c59aa55b37468fcd1262d583432dfa96fed2d8" exitCode=0
Oct 30 14:47:30.739620 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739321    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="6d3c49571aa39bb2c73d5dea20658bdfaf2ee140a524f8dc859a5464b3526e12" exitCode=0
Oct 30 14:47:30.739620 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.739326    5064 generic.go:356] "Generic (PLEG): container finished" podID="9526eb6d-d032-49ab-8a4e-901e901681a7" containerID="48b1ce7fc4deaeb7097ad571f5c537773a92db9199bfe6433a070c3c6c288c10" exitCode=0
Oct 30 14:47:30.749391 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.749359    5064 generic.go:356] "Generic (PLEG): container finished" podID="ca77c311626526aba5ca36ce028bef40" containerID="f7ba4e9ca45a549b4080db65efd14be834c0d9839c877510fd1974663e21cbdb" exitCode=0
Oct 30 14:47:30.755981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.755944    5064 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:47:30.756131 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.756074    5064 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:47:30.756586 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.756566    5064 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 14:47:30.756586 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.756578    5064 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:47:30.756877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.756851    5064 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 14:47:30.767674 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.767613    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:30.767674 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.767646    5064 generic.go:356] "Generic (PLEG): container finished" podID="f3374e02-805e-4d08-97c9-2207b67e62f8" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74" exitCode=255
Oct 30 14:47:30.791125 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.791100    5064 manager.go:324] Recovery completed
Oct 30 14:47:30.812405 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812378    5064 cpu_manager.go:226] "Starting" policy="none"
Oct 30 14:47:30.812405 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812399    5064 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812426    5064 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812559    5064 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812570    5064 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812588    5064 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812594    5064 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812599    5064 policy_none.go:50] "Start"
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812608    5064 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 14:47:30.812644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812616    5064 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 14:47:30.813001 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812740    5064 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 14:47:30.813001 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812747    5064 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 14:47:30.813001 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.812753    5064 policy_none.go:44] "Start"
Oct 30 14:47:30.814692 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:30.814660    5064 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 14:47:30.817589 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.817547    5064 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 14:47:30.820396 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.820379    5064 manager.go:341] "Starting Device Plugin manager"
Oct 30 14:47:30.822133 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:30.822106    5064 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 14:47:30.822133 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.822130    5064 server.go:87] "Starting device plugin registration server"
Oct 30 14:47:30.822430 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.822412    5064 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 14:47:30.822557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.822430    5064 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 14:47:30.823067 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.823048    5064 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 14:47:30.823191 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.823159    5064 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 14:47:30.823248 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.823201    5064 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 14:47:30.823248 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.823208    5064 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 14:47:30.823448 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.823340    5064 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:47:30.823189023Z","UUID":"1f729e26-ccdf-4c9d-84b8-9209fabb44d8","Handler":null,"Name":"","Endpoint":""}
Oct 30 14:47:30.825005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.824989    5064 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 14:47:30.825100 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.825019    5064 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 14:47:30.825575 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:30.825552    5064 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 14:47:30.923395 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.923359    5064 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 14:47:30.927322 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.927295    5064 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 14:47:30.927457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.927330    5064 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 14:47:30.927457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.927340    5064 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-143.ec2.internal" event="NodeHasSufficientPID"
Oct 30 14:47:30.927457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.927393    5064 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:47:30.969004 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.968973    5064 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:47:30.969155 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:30.969062    5064 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.015524 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.015474    5064 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"]
Oct 30 14:47:31.016116 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016091    5064 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.016116 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016086    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal" event={"ID":"d6501707c434eeb14bc7e15cb32ecb7b","Type":"ContainerStarted","Data":"2d05b80b7af45e8a4de302278835f1f86b4aa8d30082d3f1f303f812301ca2bb"}
Oct 30 14:47:31.016323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016197    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal" event={"ID":"d6501707c434eeb14bc7e15cb32ecb7b","Type":"ContainerStarted","Data":"6ed0b99d42d5edcd4ad8f7bc2813fb73c5791a65f39edc2e5be69e42a780cd71"}
Oct 30 14:47:31.016323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016214    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerStarted","Data":"97766a9df919711e30ba24246fcbd5d2c01cfa7c75bfe926fc0493728ef67ba3"}
Oct 30 14:47:31.016323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016227    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerDied","Data":"f7ba4e9ca45a549b4080db65efd14be834c0d9839c877510fd1974663e21cbdb"}
Oct 30 14:47:31.016323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016247    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal" event={"ID":"ca77c311626526aba5ca36ce028bef40","Type":"ContainerStarted","Data":"16cf5f77f59137173b3d4f282cbf1e1d67f7a16eaba4081135b62da959a5de20"}
Oct 30 14:47:31.016517 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.016337    5064 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.042706 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.042630    5064 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:47:31.042706 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.042694    5064 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.044189 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.044169    5064 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:47:31.044232 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.044219    5064 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.157908 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.157874    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.157908 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.157908    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.158129 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.157928    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258643 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258609    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258655    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258675    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258721    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258735    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca77c311626526aba5ca36ce028bef40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" (UID: \"ca77c311626526aba5ca36ce028bef40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.258814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.258798    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6501707c434eeb14bc7e15cb32ecb7b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" (UID: \"d6501707c434eeb14bc7e15cb32ecb7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.662691 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.662651    5064 apiserver.go:52] "Watching apiserver"
Oct 30 14:47:31.722699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.722664    5064 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 14:47:31.723522 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.723488    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9849b","openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m","openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg","kube-system/kas-connection-checker-685895c7c9-tlhw9","openshift-network-console/networking-console-plugin-86c478bb88-m9l7n","openshift-multus/multus-additional-cni-plugins-9vztw","openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77","openshift-ingress/router-default-58ddf469d5-lvlsc","openshift-network-operator/iptables-alerter-76w97","openshift-service-ca/service-ca-5fffc97bdf-pg7ql","openshift-cluster-node-tuning-operator/tuned-tp4ll","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs","openshift-network-diagnostics/network-check-target-zxt6d","kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal","openshift-dns/dns-default-2fr4t","kube-system/konnectivity-agent-k4c5w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal","kube-system/kas-connection-checker-685895c7c9-kfr99","openshift-image-registry/image-registry-766565bc5f-b7l2r","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm","openshift-insights/insights-operator-7c4986d449-njl5k","openshift-ovn-kubernetes/ovnkube-node-kd44p","openshift-dns/node-resolver-vnzrb","openshift-multus/network-metrics-daemon-grqw8","openshift-insights/periodic-gathering-dzxvg-c8zl2","kube-system/global-pull-secret-syncer-pnhqm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd","openshift-image-registry/image-registry-b65c48686-qftvr","openshift-multus/multus-rf9g5","kube-system/kas-connection-checker-685895c7c9-lvqzv","openshift-console-operator/console-operator-7cd56b8dbc-7j8cn","openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm","openshift-image-registry/node-ca-96lz7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588","openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz","openshift-insights/insights-runtime-extractor-s4n86"]
Oct 30 14:47:31.725116 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.725092    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.726549 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.726530    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.727206 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.727188    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.727895 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.727878    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.728337 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.728320    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.728751 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.728732    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.737031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.737005    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:31.739949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.739930    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.740319 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.740295    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.740417 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.740303    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.741566 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.741548    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.741998 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.741981    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.742455 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742436    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 14:47:31.742679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742657    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-slvwr\""
Oct 30 14:47:31.742767 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742672    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5vnjn\""
Oct 30 14:47:31.742767 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742719    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 14:47:31.742767 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742723    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.742936 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742787    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g9j6g\""
Oct 30 14:47:31.742936 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742910    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 14:47:31.743017 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742989    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 14:47:31.743017 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.742995    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.743105 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743020    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.743105 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743056    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 14:47:31.743213 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743104    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 14:47:31.743280 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743259    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l2gbg\""
Oct 30 14:47:31.743359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743306    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 14:47:31.743359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.743330    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.756752 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756721    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wbwnx\""
Oct 30 14:47:31.756752 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756753    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 14:47:31.756987 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756755    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.756987 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756720    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 14:47:31.756987 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756770    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.756987 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.756781    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.757250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757034    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 14:47:31.757250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757123    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 14:47:31.757250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757157    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.757250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757190    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.757250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757203    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.757848 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757805    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 14:47:31.757848 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757848    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 14:47:31.757976 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757850    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.757976 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757894    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.758113 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.757977    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bhrs9\""
Oct 30 14:47:31.758113 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758026    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 14:47:31.758113 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758057    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.758362 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758347    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 14:47:31.758874 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758822    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 14:47:31.758990 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758917    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 14:47:31.758990 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.758940    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8pjkc\""
Oct 30 14:47:31.759478 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.759454    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nh5lx\""
Oct 30 14:47:31.759623 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.759607    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 14:47:31.759846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.759811    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nrd46\""
Oct 30 14:47:31.760335 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.760302    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 14:47:31.760406 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.760058    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 14:47:31.760406 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.759871    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qts79\""
Oct 30 14:47:31.760406 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.760292    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\""
Oct 30 14:47:31.760534 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.760151    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 14:47:31.770381 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770356    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770445    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 14:47:31.770491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770466    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770478    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.770491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770488    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770501    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770506    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770527    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770534    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770581    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770651    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.770697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770692    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770762    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770871    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770920    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770931    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770940    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770969    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.770997    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\""
Oct 30 14:47:31.771025 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771005    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771035    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771078    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771130    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771135    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771243    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771281    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771281    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771373    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771382    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771386    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9fbb\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771405    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771409    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 14:47:31.771432 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771427    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771476    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771502    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771511    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771526    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771504    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771527    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771527    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771556    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771569    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771574    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771576    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771610    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771618    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771641    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771652    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xtzxg\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771652    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hg7s\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771666    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771700    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771729    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771758    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771769    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771804    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771813    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771761    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771848    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771864    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771897    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771897    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 14:47:31.772005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.771989    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 14:47:31.772883 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.772058    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\""
Oct 30 14:47:31.772883 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.772174    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\""
Oct 30 14:47:31.772883 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.772237    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 14:47:31.772883 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.772295    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 14:47:31.776949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.776932    5064 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.777078 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.777065    5064 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.781144 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.781128    5064 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 14:47:31.801756 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.801733    5064 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:47:31.801869 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.801791    5064 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.801869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.801734    5064 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 14:47:31.801979 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.801878    5064 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-143.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-143.ec2.internal"
Oct 30 14:47:31.863317 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863282    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflr4\" (UniqueName: \"kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4\") pod \"migrator-66df5d496b-xgd8m\" (UID: \"4f07f4f5-11c1-4474-8303-3f7cbd171a7d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:31.863317 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863314    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh59\" (UniqueName: \"kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.863540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863335    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.863540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863361    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.863540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863386    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.863540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863477    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:31.863744 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863564    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.863744 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863603    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vpn\" (UniqueName: \"kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.863744 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863656    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.863744 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863704    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863752    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863786    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863801    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-konnectivity-ca\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863817    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863888    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863946    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.863981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.863979    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864017    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864056    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864082    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864102    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864110    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864154    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsrc\" (UniqueName: \"kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864194    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864198    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d22383-d225-4613-baf3-30a0b288519e-tmp-dir\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864214    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-config\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864223    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-tmp\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.864240 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864234    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wft\" (UniqueName: \"kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864259    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt98x\" (UniqueName: \"kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864283    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864310    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864333    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864366    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-cni-binary-copy\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864357    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864390    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-daemon-config\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864446    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864567    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b767b7e7-8048-4086-aa1b-5ec54e3d0568-serving-cert\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.864603 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864589    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864628    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a72ab8d-d509-46b8-809f-55a708439ffd-original-pull-secret\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864637    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864668    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864705    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864768    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864795    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864816    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864909    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/107da892-9033-4172-b313-2431ef06d16c-serving-cert\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.865166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864975    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.864889    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865325    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865347    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865365    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865390    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2zt\" (UniqueName: \"kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865431    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865441    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3374e02-805e-4d08-97c9-2207b67e62f8-trusted-ca\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865480    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-script-lib\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.865548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865547    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865566    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865633    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865672    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865697    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpr6\" (UniqueName: \"kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865715    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865773    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp8v\" (UniqueName: \"kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v\") pod \"kas-connection-checker-685895c7c9-kfr99\" (UID: \"26527e26-b484-446f-b704-ee332fea2f2e\") " pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865783    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.865817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865807    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865861    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865890    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865914    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865940    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzwc\" (UniqueName: \"kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865965    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865985    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-tmp\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865991    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.865986    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-tmp-dir\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866093    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866122    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdx7\" (UniqueName: \"kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866150    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.866188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866169    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866196    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866234    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866372    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43cc708-9756-48e8-a788-a79a9a016e9a-data-volume\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866378    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866374    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3374e02-805e-4d08-97c9-2207b67e62f8-serving-cert\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866412    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866441    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gzn\" (UniqueName: \"kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866470    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866497    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866523    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866577    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866591    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-config-volume\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866604    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm2z\" (UniqueName: \"kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866631    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866635    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfad0d99-650b-4984-8afb-757f92bf7db9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866675    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866701    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.866733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866728    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866751    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866776    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866801    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866870    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866883    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-default-certificate\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866894    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3-agent-certs\") pod \"konnectivity-agent-k4c5w\" (UID: \"c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3\") " pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866908    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866939    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.866954    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867001    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867016    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frz62\" (UniqueName: \"kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867036    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867038    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-key\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867075    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09ec3816-9582-4e08-b9d0-9d03174f5628-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867131    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867171    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867253    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867293    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscxx\" (UniqueName: \"kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx\") pod \"migrator-66df5d496b-7qshm\" (UID: \"057216e1-88ce-443d-8033-657bfc04ad17\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867325    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867293    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867353    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqqn\" (UniqueName: \"kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867413    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-env-overrides\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867488    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vprm\" (UniqueName: \"kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm\") pod \"volume-data-source-validator-58c9f5f46d-qdfhs\" (UID: \"30c328c3-1bca-4576-9fd0-c669059e423e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867521    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867549    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zttn\" (UniqueName: \"kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.867574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867580    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867612    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867661    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867720    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr2p\" (UniqueName: \"kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867767    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867799    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867861    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867893    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmksh\" (UniqueName: \"kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh\") pod \"kas-connection-checker-685895c7c9-tlhw9\" (UID: \"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8\") " pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867949    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867945    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-cert\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.867987    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5k9\" (UniqueName: \"kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9\") pod \"network-check-source-6f78d59f99-k5zrz\" (UID: \"ba2f7212-f693-4d67-8b61-fc75f1ad0372\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868017    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868051    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868076    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868139    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868160    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbr89\" (UniqueName: \"kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868175    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-serviceca\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868196    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868224    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868253    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868320    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868360    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b767b7e7-8048-4086-aa1b-5ec54e3d0568-config\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868365    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868391    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868415    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868445    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868507    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.868582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868547    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868600    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868643    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln786\" (UniqueName: \"kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868688    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868714    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868715    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/107da892-9033-4172-b313-2431ef06d16c-service-ca-bundle\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868773    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/107da892-9033-4172-b313-2431ef06d16c-snapshots\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868795    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868809    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868863    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868941    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.868990    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-stats-auth\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869079    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869081    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79986431-390d-4313-af30-b2ff62e0ce68-etc-tuned\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869198    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869218    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869232    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869389    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71d35a9c-d95f-4eba-a095-1f95b931f16a-iptables-alerter-script\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869392    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpt7\" (UniqueName: \"kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869454    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869493    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869528    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869560    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.869588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869592    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869628    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869664    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmgx\" (UniqueName: \"kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869695    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869730    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9nn\" (UniqueName: \"kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869759    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869795    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxdr\" (UniqueName: \"kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr\") pod \"kas-connection-checker-685895c7c9-lvqzv\" (UID: \"ebdd075b-3f66-4f77-ab73-e8f080e78f37\") " pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869843    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869876    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869909    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869936    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869994    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.869992    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea952af6-f7f1-43d6-82b4-8137b837a4e0-ovnkube-config\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870050    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870069    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870107    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870138    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870140    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9526eb6d-d032-49ab-8a4e-901e901681a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870143    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870263    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870377    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870421    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870488    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870519    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870544    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870588    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.870619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870612    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.870635    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.871001    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4dd5b57e-2745-4bc1-8331-abac3f327a13-signing-cabundle\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.871033    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.871132    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09ec3816-9582-4e08-b9d0-9d03174f5628-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.871315    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d375dcba-a3e9-40ec-9e41-fb7e692430f4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.871509 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.871432    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.886889 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.886861    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh59\" (UniqueName: \"kubernetes.io/projected/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-kube-api-access-lhh59\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.887482 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.887466    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflr4\" (UniqueName: \"kubernetes.io/projected/4f07f4f5-11c1-4474-8303-3f7cbd171a7d-kube-api-access-kflr4\") pod \"migrator-66df5d496b-xgd8m\" (UID: \"4f07f4f5-11c1-4474-8303-3f7cbd171a7d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xgd8m"
Oct 30 14:47:31.889775 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.889758    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wft\" (UniqueName: \"kubernetes.io/projected/79986431-390d-4313-af30-b2ff62e0ce68-kube-api-access-v7wft\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.891110 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.891092    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsrc\" (UniqueName: \"kubernetes.io/projected/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-api-access-zfsrc\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.891221 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.891201    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt98x\" (UniqueName: \"kubernetes.io/projected/43d22383-d225-4613-baf3-30a0b288519e-kube-api-access-qt98x\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.894812 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.894795    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vpn\" (UniqueName: \"kubernetes.io/projected/f3374e02-805e-4d08-97c9-2207b67e62f8-kube-api-access-f8vpn\") pod \"console-operator-7cd56b8dbc-7j8cn\" (UID: \"f3374e02-805e-4d08-97c9-2207b67e62f8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:31.898926 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.898906    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmksh\" (UniqueName: \"kubernetes.io/projected/ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8-kube-api-access-bmksh\") pod \"kas-connection-checker-685895c7c9-tlhw9\" (UID: \"ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8\") " pod="kube-system/kas-connection-checker-685895c7c9-tlhw9"
Oct 30 14:47:31.904880 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.904782    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln786\" (UniqueName: \"kubernetes.io/projected/09ec3816-9582-4e08-b9d0-9d03174f5628-kube-api-access-ln786\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s5djm\" (UID: \"09ec3816-9582-4e08-b9d0-9d03174f5628\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm"
Oct 30 14:47:31.908127 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.905149    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr2p\" (UniqueName: \"kubernetes.io/projected/b7f9e692-eeb3-4000-80dc-0eebeb13f1db-kube-api-access-bwr2p\") pod \"ingress-canary-9849b\" (UID: \"b7f9e692-eeb3-4000-80dc-0eebeb13f1db\") " pod="openshift-ingress-canary/ingress-canary-9849b"
Oct 30 14:47:31.908127 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.905424    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vprm\" (UniqueName: \"kubernetes.io/projected/30c328c3-1bca-4576-9fd0-c669059e423e-kube-api-access-6vprm\") pod \"volume-data-source-validator-58c9f5f46d-qdfhs\" (UID: \"30c328c3-1bca-4576-9fd0-c669059e423e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qdfhs"
Oct 30 14:47:31.910542 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.910515    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frz62\" (UniqueName: \"kubernetes.io/projected/b767b7e7-8048-4086-aa1b-5ec54e3d0568-kube-api-access-frz62\") pod \"service-ca-operator-544d749c8c-tfwwg\" (UID: \"b767b7e7-8048-4086-aa1b-5ec54e3d0568\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg"
Oct 30 14:47:31.910658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.910562    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpr6\" (UniqueName: \"kubernetes.io/projected/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-kube-api-access-bcpr6\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.910723 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.910703    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"periodic-gathering-dzxvg-c8zl2\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") " pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:47:31.910913 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.910893    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm2z\" (UniqueName: \"kubernetes.io/projected/71d35a9c-d95f-4eba-a095-1f95b931f16a-kube-api-access-jvm2z\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.912330 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912304    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmgx\" (UniqueName: \"kubernetes.io/projected/4dd5b57e-2745-4bc1-8331-abac3f327a13-kube-api-access-wlmgx\") pod \"service-ca-5fffc97bdf-pg7ql\" (UID: \"4dd5b57e-2745-4bc1-8331-abac3f327a13\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pg7ql"
Oct 30 14:47:31.912330 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912319    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqqn\" (UniqueName: \"kubernetes.io/projected/dee6e3a3-595c-4f16-a095-4e79aae97230-kube-api-access-tzqqn\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.912461 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912334    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscxx\" (UniqueName: \"kubernetes.io/projected/057216e1-88ce-443d-8033-657bfc04ad17-kube-api-access-sscxx\") pod \"migrator-66df5d496b-7qshm\" (UID: \"057216e1-88ce-443d-8033-657bfc04ad17\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qshm"
Oct 30 14:47:31.912542 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912522    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2zt\" (UniqueName: \"kubernetes.io/projected/0badeea4-6775-4cf7-ad74-887fa9ee375a-kube-api-access-kx2zt\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.912669 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912652    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5k9\" (UniqueName: \"kubernetes.io/projected/ba2f7212-f693-4d67-8b61-fc75f1ad0372-kube-api-access-hh5k9\") pod \"network-check-source-6f78d59f99-k5zrz\" (UID: \"ba2f7212-f693-4d67-8b61-fc75f1ad0372\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k5zrz"
Oct 30 14:47:31.912758 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912719    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpt7\" (UniqueName: \"kubernetes.io/projected/d375dcba-a3e9-40ec-9e41-fb7e692430f4-kube-api-access-kwpt7\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.912936 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.912908    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zttn\" (UniqueName: \"kubernetes.io/projected/107da892-9033-4172-b313-2431ef06d16c-kube-api-access-9zttn\") pod \"insights-operator-7c4986d449-njl5k\" (UID: \"107da892-9033-4172-b313-2431ef06d16c\") " pod="openshift-insights/insights-operator-7c4986d449-njl5k"
Oct 30 14:47:31.914993 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.914972    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzwc\" (UniqueName: \"kubernetes.io/projected/4ae91844-19fb-4008-89ce-5e427270dc96-kube-api-access-rvzwc\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.915066 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.915027    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.915161 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.915121    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbr89\" (UniqueName: \"kubernetes.io/projected/82e0ad12-29e4-4892-841b-801edb30b567-kube-api-access-nbr89\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:31.915437 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.915421    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.915500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.915486    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gzn\" (UniqueName: \"kubernetes.io/projected/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-kube-api-access-q2gzn\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.915882 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.915861    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxdr\" (UniqueName: \"kubernetes.io/projected/ebdd075b-3f66-4f77-ab73-e8f080e78f37-kube-api-access-pgxdr\") pod \"kas-connection-checker-685895c7c9-lvqzv\" (UID: \"ebdd075b-3f66-4f77-ab73-e8f080e78f37\") " pod="kube-system/kas-connection-checker-685895c7c9-lvqzv"
Oct 30 14:47:31.916049 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.916032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9nn\" (UniqueName: \"kubernetes.io/projected/9526eb6d-d032-49ab-8a4e-901e901681a7-kube-api-access-wp9nn\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.916504 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.916483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.917439 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.917423    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp8v\" (UniqueName: \"kubernetes.io/projected/26527e26-b484-446f-b704-ee332fea2f2e-kube-api-access-lwp8v\") pod \"kas-connection-checker-685895c7c9-kfr99\" (UID: \"26527e26-b484-446f-b704-ee332fea2f2e\") " pod="kube-system/kas-connection-checker-685895c7c9-kfr99"
Oct 30 14:47:31.917554 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.917540    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.917709 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.917691    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgzb\" (UniqueName: \"kubernetes.io/projected/b12b5bea-cee8-4db4-89a3-ddec80c8f82a-kube-api-access-2kgzb\") pod \"network-check-target-zxt6d\" (UID: \"b12b5bea-cee8-4db4-89a3-ddec80c8f82a\") " pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:31.921163 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.921145    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdx7\" (UniqueName: \"kubernetes.io/projected/ea952af6-f7f1-43d6-82b4-8137b837a4e0-kube-api-access-bzdx7\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971270    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971310    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971336    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971358    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971413    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-cnibin\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971413    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-etc-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971433    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971437    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43d22383-d225-4613-baf3-30a0b288519e-hosts-file\") pod \"node-resolver-vnzrb\" (UID: \"43d22383-d225-4613-baf3-30a0b288519e\") " pod="openshift-dns/node-resolver-vnzrb"
Oct 30 14:47:31.971485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971472    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971477    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-conf\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971471    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-bin\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971502    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971543    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971550    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-socket-dir-parent\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971568    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971590    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971606    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971634    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971640    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-slash\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971657    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971669    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-multus\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971694    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-systemd\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971695    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971730    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971729    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-cni-bin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971752    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-log-socket\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971765    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971769    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-kubelet\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971792    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971811    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-systemd-units\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971815    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-var-lib-kubelet\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971818    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971901    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971900    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971883    5064 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971945    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-system-cni-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.971944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971932    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971993    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972018    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972038    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-lib-modules\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972043    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.971962    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-systemd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972077    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-host\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972103    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-dbus\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972107    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972130    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972160    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972184    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972207    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972235    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972248    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-device-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972279    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972290    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4-host\") pod \"node-ca-96lz7\" (UID: \"9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4\") " pod="openshift-image-registry/node-ca-96lz7"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972309    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972336    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972359    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972383    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972408    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.972424    5064 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972435    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972463    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972489    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.972510    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:32.472487506 +0000 UTC m=+1.914378221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : secret "metrics-daemon-secret" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972520    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-var-lib-openvswitch\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972544    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972570    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-k8s-cni-cncf-io\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972578    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972617    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972633    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9526eb6d-d032-49ab-8a4e-901e901681a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vztw\" (UID: \"9526eb6d-d032-49ab-8a4e-901e901681a7\") " pod="openshift-multus/multus-additional-cni-plugins-9vztw"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972645    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972675    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972685    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysconfig\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.972704    5064 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972732    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f43cc708-9756-48e8-a788-a79a9a016e9a-crio-socket\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.972749    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:32.472733405 +0000 UTC m=+1.914624115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972702    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972756    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-node-log\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972771    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-netns\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972780    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972808    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972870    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972910    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972923    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-sysctl-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972967    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d35a9c-d95f-4eba-a095-1f95b931f16a-host-slash\") pod \"iptables-alerter-76w97\" (UID: \"71d35a9c-d95f-4eba-a095-1f95b931f16a\") " pod="openshift-network-operator/iptables-alerter-76w97"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.972971    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-host-run-multus-certs\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.973002    5064 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973015    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.973059    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:32.473042692 +0000 UTC m=+1.914933400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973084    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-sys-fs\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973088    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a72ab8d-d509-46b8-809f-55a708439ffd-kubelet-config\") pod \"global-pull-secret-syncer-pnhqm\" (UID: \"9a72ab8d-d509-46b8-809f-55a708439ffd\") " pod="kube-system/global-pull-secret-syncer-pnhqm"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973151    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973131    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-netns\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973218    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973244    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973270    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.973249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973282    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f43cc708-9756-48e8-a788-a79a9a016e9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973301    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973345    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973353    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-run\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973354    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-system-cni-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973318    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-socket-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973372    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973395    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-etc-kubernetes\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973419    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973424    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-registration-dir\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973433    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-run-ovn\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0badeea4-6775-4cf7-ad74-887fa9ee375a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8snsd\" (UID: \"0badeea4-6775-4cf7-ad74-887fa9ee375a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8snsd"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973541    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-service-ca-bundle\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973547    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973450    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-sys\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973624    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973652    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973677    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973708    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973733    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973757    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973780    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973803    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973847    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973873    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.973966    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-kubernetes\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974023    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-var-lib-kubelet\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974035    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-cni-netd\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974063    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-hostroot\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974083    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-cnibin\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974102    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dee6e3a3-595c-4f16-a095-4e79aae97230-multus-conf-dir\") pod \"multus-rf9g5\" (UID: \"dee6e3a3-595c-4f16-a095-4e79aae97230\") " pod="openshift-multus/multus-rf9g5"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974169    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea952af6-f7f1-43d6-82b4-8137b837a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-kd44p\" (UID: \"ea952af6-f7f1-43d6-82b4-8137b837a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974215    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79986431-390d-4313-af30-b2ff62e0ce68-etc-modprobe-d\") pod \"tuned-tp4ll\" (UID: \"79986431-390d-4313-af30-b2ff62e0ce68\") " pod="openshift-cluster-node-tuning-operator/tuned-tp4ll"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.974771    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e34f75c-8c90-48eb-b5e6-4d363017a5f3-metrics-certs\") pod \"router-default-58ddf469d5-lvlsc\" (UID: \"9e34f75c-8c90-48eb-b5e6-4d363017a5f3\") " pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.975256    5064 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.975922    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"image-registry-b65c48686-qftvr\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") " pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:31.976104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.975922    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfad0d99-650b-4984-8afb-757f92bf7db9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m9l7n\" (UID: \"bfad0d99-650b-4984-8afb-757f92bf7db9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:31.977274 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.976343    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"image-registry-766565bc5f-b7l2r\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") " pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:31.977274 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:31.976849    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ae91844-19fb-4008-89ce-5e427270dc96-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v4588\" (UID: \"4ae91844-19fb-4008-89ce-5e427270dc96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:31.977274 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:31.976871    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:32.47629624 +0000 UTC m=+1.918186959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:32.025800 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.025769    5064 scope.go:122] "RemoveContainer" containerID="2ac8532c89b06dc843bc2826627348895ece3bf92bea88de08e2ad79574f1d74"
Oct 30 14:47:32.025958 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.025850    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"
Oct 30 14:47:32.027568 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.027551    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:32.028222 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.028206    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:32.028564 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.028550    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"
Oct 30 14:47:32.029170 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.029157    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:32.116381 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.116349    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:32.281776 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.281705    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:32.356963 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.356935    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588"]
Oct 30 14:47:32.374856 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.374224    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m9l7n"]
Oct 30 14:47:32.378958 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:32.378921    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfad0d99_650b_4984_8afb_757f92bf7db9.slice/crio-d0b7f020001dcb83fc7f419bc2acd7815abfdd81e3c3720d7cc13ae4419ee4fc WatchSource:0}: Error finding container d0b7f020001dcb83fc7f419bc2acd7815abfdd81e3c3720d7cc13ae4419ee4fc: Status 404 returned error can't find the container with id d0b7f020001dcb83fc7f419bc2acd7815abfdd81e3c3720d7cc13ae4419ee4fc
Oct 30 14:47:32.381796 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.381774    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:47:32.381914 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.381882    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:47:32.385401 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:32.385376    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbee651d_49b7_4013_b903_da36898913cf.slice/crio-439bf64a229cbc532e15768f15370156202cc95f330ca494ada3639c41921184 WatchSource:0}: Error finding container 439bf64a229cbc532e15768f15370156202cc95f330ca494ada3639c41921184: Status 404 returned error can't find the container with id 439bf64a229cbc532e15768f15370156202cc95f330ca494ada3639c41921184
Oct 30 14:47:32.396775 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.395931    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:47:32.398624 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.398603    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58ddf469d5-lvlsc"]
Oct 30 14:47:32.398980 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:32.398953    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8407bb01_c8e0_4624_bd80_c27459b6c96e.slice/crio-8cc5c7a1f2d3deafca3b6f5a68db6a1602e20428fb5f9dba979e5ffc17557ed6 WatchSource:0}: Error finding container 8cc5c7a1f2d3deafca3b6f5a68db6a1602e20428fb5f9dba979e5ffc17557ed6: Status 404 returned error can't find the container with id 8cc5c7a1f2d3deafca3b6f5a68db6a1602e20428fb5f9dba979e5ffc17557ed6
Oct 30 14:47:32.405228 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:32.405200    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e34f75c_8c90_48eb_b5e6_4d363017a5f3.slice/crio-dc7bfac3c99ce86310ae9d93db80157cb50fdbf3fc0ced9c77b66facc1da7535 WatchSource:0}: Error finding container dc7bfac3c99ce86310ae9d93db80157cb50fdbf3fc0ced9c77b66facc1da7535: Status 404 returned error can't find the container with id dc7bfac3c99ce86310ae9d93db80157cb50fdbf3fc0ced9c77b66facc1da7535
Oct 30 14:47:32.428329 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.428290    5064 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:32.478485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.478458    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:32.478628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.478532    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:32.478628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.478561    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:32.478628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.478584    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478625    5064 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478674    5064 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478695    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.478672029 +0000 UTC m=+2.920562726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478739    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.478703659 +0000 UTC m=+2.920594354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478759    5064 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478785    5064 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:32.478800 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478790    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.478780256 +0000 UTC m=+2.920670954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : secret "metrics-daemon-secret" not found
Oct 30 14:47:32.479200 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:32.478811    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:33.478804362 +0000 UTC m=+2.920695057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:32.783225 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.783187    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" event={"ID":"cbee651d-49b7-4013-b903-da36898913cf","Type":"ContainerStarted","Data":"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"}
Oct 30 14:47:32.783700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.783264    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" event={"ID":"cbee651d-49b7-4013-b903-da36898913cf","Type":"ContainerStarted","Data":"439bf64a229cbc532e15768f15370156202cc95f330ca494ada3639c41921184"}
Oct 30 14:47:32.785935 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.785869    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588" event={"ID":"4ae91844-19fb-4008-89ce-5e427270dc96","Type":"ContainerStarted","Data":"4c80fca314b589836c7996e82d34c3a0fed6b36cd39f3d3a3cdbf45c3d78416d"}
Oct 30 14:47:32.788279 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.788251    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:47:32.788547 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.788481    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn" event={"ID":"f3374e02-805e-4d08-97c9-2207b67e62f8","Type":"ContainerStarted","Data":"e69ad6d320606ad922147345dcea844f7cc8b62190772c589330ab9760797e30"}
Oct 30 14:47:32.789030 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.788906    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:32.797586 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.797560    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7j8cn"
Oct 30 14:47:32.800184 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.800128    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n" event={"ID":"bfad0d99-650b-4984-8afb-757f92bf7db9","Type":"ContainerStarted","Data":"d0b7f020001dcb83fc7f419bc2acd7815abfdd81e3c3720d7cc13ae4419ee4fc"}
Oct 30 14:47:32.812858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.806518    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b65c48686-qftvr" event={"ID":"8407bb01-c8e0-4624-bd80-c27459b6c96e","Type":"ContainerStarted","Data":"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"}
Oct 30 14:47:32.812858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.806563    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b65c48686-qftvr" event={"ID":"8407bb01-c8e0-4624-bd80-c27459b6c96e","Type":"ContainerStarted","Data":"8cc5c7a1f2d3deafca3b6f5a68db6a1602e20428fb5f9dba979e5ffc17557ed6"}
Oct 30 14:47:32.812858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.811159    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58ddf469d5-lvlsc" event={"ID":"9e34f75c-8c90-48eb-b5e6-4d363017a5f3","Type":"ContainerStarted","Data":"0a2ef9dd7dd8549ac10000b0161f1af6fa007fdcc1815004d6942ae5073c19b4"}
Oct 30 14:47:32.812858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:32.811194    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58ddf469d5-lvlsc" event={"ID":"9e34f75c-8c90-48eb-b5e6-4d363017a5f3","Type":"ContainerStarted","Data":"dc7bfac3c99ce86310ae9d93db80157cb50fdbf3fc0ced9c77b66facc1da7535"}
Oct 30 14:47:33.028449 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.028411    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:33.031104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.031078    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:33.389567 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.389527    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:33.390906 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.390880    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58ddf469d5-lvlsc"
Oct 30 14:47:33.492043 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.492017    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:33.492173 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.492062    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:33.492173 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.492151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:33.492276 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492174    5064 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:33.492276 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.492238    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:33.492276 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492266    5064 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 14:47:33.492419 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492305    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls podName:d375dcba-a3e9-40ec-9e41-fb7e692430f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:35.492288476 +0000 UTC m=+4.934179177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dtg77" (UID: "d375dcba-a3e9-40ec-9e41-fb7e692430f4") : secret "cluster-monitoring-operator-tls" not found
Oct 30 14:47:33.492419 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492325    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs podName:82e0ad12-29e4-4892-841b-801edb30b567 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:35.492314399 +0000 UTC m=+4.934205099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs") pod "network-metrics-daemon-grqw8" (UID: "82e0ad12-29e4-4892-841b-801edb30b567") : secret "metrics-daemon-secret" not found
Oct 30 14:47:33.492526 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492347    5064 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:33.492526 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492503    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:35.492493737 +0000 UTC m=+4.934384444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:33.492526 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492385    5064 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 14:47:33.492647 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:33.492543    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls podName:bef2ba8e-37ad-41c2-a322-b481ba3be1e1 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:35.492534237 +0000 UTC m=+4.934424949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls") pod "dns-default-2fr4t" (UID: "bef2ba8e-37ad-41c2-a322-b481ba3be1e1") : secret "dns-default-metrics-tls" not found
Oct 30 14:47:33.816749 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.816676    5064 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:33.817207 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.816915    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m9l7n" event={"ID":"bfad0d99-650b-4984-8afb-757f92bf7db9","Type":"ContainerStarted","Data":"ed8101224ce0b75198943e85641844af2e530f9191df2618042132fc3778805e"}
Oct 30 14:47:33.817356 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:33.817339    5064 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:34.408345 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:34.408312    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:47:34.823009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:34.822912    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588" event={"ID":"4ae91844-19fb-4008-89ce-5e427270dc96","Type":"ContainerStarted","Data":"b7d107fd679e9f7983347e94e7eb57539cea6e7ba6792c41bc0ff7fa8678fa66"}
Oct 30 14:47:34.823009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:34.822980    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v4588" event={"ID":"4ae91844-19fb-4008-89ce-5e427270dc96","Type":"ContainerStarted","Data":"0236774a22d3a41c59a6352ae477c44562ca840d640d5bfb5f83cb05ac300770"}
Oct 30 14:47:35.511266 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.511222    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:35.511459 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.511298    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:35.511459 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.511320    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:35.511459 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.511339    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:35.511459 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:35.511442    5064 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:35.511669 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:35.511544    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls podName:f43cc708-9756-48e8-a788-a79a9a016e9a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:39.511524591 +0000 UTC m=+8.953415300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n86" (UID: "f43cc708-9756-48e8-a788-a79a9a016e9a") : secret "insights-runtime-extractor-tls" not found
Oct 30 14:47:35.513866 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.513821    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef2ba8e-37ad-41c2-a322-b481ba3be1e1-metrics-tls\") pod \"dns-default-2fr4t\" (UID: \"bef2ba8e-37ad-41c2-a322-b481ba3be1e1\") " pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:35.514009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.513823    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e0ad12-29e4-4892-841b-801edb30b567-metrics-certs\") pod \"network-metrics-daemon-grqw8\" (UID: \"82e0ad12-29e4-4892-841b-801edb30b567\") " pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:35.514009 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.513931    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d375dcba-a3e9-40ec-9e41-fb7e692430f4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dtg77\" (UID: \"d375dcba-a3e9-40ec-9e41-fb7e692430f4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:35.627899 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.627859    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"
Oct 30 14:47:35.638353 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.638331    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-grqw8"
Oct 30 14:47:35.640513 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.640484    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:35.828258 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.828165    5064 generic.go:356] "Generic (PLEG): container finished" podID="107da892-9033-4172-b313-2431ef06d16c" containerID="856c8dca650f9f0197a0c3b97999e1cc9ff1541f48a65a77a8cf43985995eaf7" exitCode=0
Oct 30 14:47:35.828258 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.828244    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerDied","Data":"856c8dca650f9f0197a0c3b97999e1cc9ff1541f48a65a77a8cf43985995eaf7"}
Oct 30 14:47:35.828712 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.828699    5064 scope.go:122] "RemoveContainer" containerID="856c8dca650f9f0197a0c3b97999e1cc9ff1541f48a65a77a8cf43985995eaf7"
Oct 30 14:47:35.946611 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.946577    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77"]
Oct 30 14:47:35.950260 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:35.950230    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd375dcba_a3e9_40ec_9e41_fb7e692430f4.slice/crio-7c4188d30a2443b7e7ab12624305d90cea5f9e3624e77f688d0e59ecffd48b3e WatchSource:0}: Error finding container 7c4188d30a2443b7e7ab12624305d90cea5f9e3624e77f688d0e59ecffd48b3e: Status 404 returned error can't find the container with id 7c4188d30a2443b7e7ab12624305d90cea5f9e3624e77f688d0e59ecffd48b3e
Oct 30 14:47:35.950395 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.950378    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fr4t"]
Oct 30 14:47:35.951604 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:35.951578    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-grqw8"]
Oct 30 14:47:35.954255 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:35.954225    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e0ad12_29e4_4892_841b_801edb30b567.slice/crio-623e42c88f59d209ee801eac3f1a624970ca8187b14a52ab98bffc0cacc93249 WatchSource:0}: Error finding container 623e42c88f59d209ee801eac3f1a624970ca8187b14a52ab98bffc0cacc93249: Status 404 returned error can't find the container with id 623e42c88f59d209ee801eac3f1a624970ca8187b14a52ab98bffc0cacc93249
Oct 30 14:47:35.956270 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:35.956233    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef2ba8e_37ad_41c2_a322_b481ba3be1e1.slice/crio-41333b67379b653a5de12e7cb687e6eba89711041144bc021f415b43f8b16368 WatchSource:0}: Error finding container 41333b67379b653a5de12e7cb687e6eba89711041144bc021f415b43f8b16368: Status 404 returned error can't find the container with id 41333b67379b653a5de12e7cb687e6eba89711041144bc021f415b43f8b16368
Oct 30 14:47:36.035172 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.035141    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:36.035422 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.035406    5064 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 14:47:36.837144 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.837063    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77" event={"ID":"d375dcba-a3e9-40ec-9e41-fb7e692430f4","Type":"ContainerStarted","Data":"7c4188d30a2443b7e7ab12624305d90cea5f9e3624e77f688d0e59ecffd48b3e"}
Oct 30 14:47:36.838751 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.838686    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-grqw8" event={"ID":"82e0ad12-29e4-4892-841b-801edb30b567","Type":"ContainerStarted","Data":"623e42c88f59d209ee801eac3f1a624970ca8187b14a52ab98bffc0cacc93249"}
Oct 30 14:47:36.841232 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.841193    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerStarted","Data":"19ebbae3e427b4be8dcb15461437930a8d9f2255a65fe84d072d58503e616e58"}
Oct 30 14:47:36.843429 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.843403    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fr4t" event={"ID":"bef2ba8e-37ad-41c2-a322-b481ba3be1e1","Type":"ContainerStarted","Data":"41333b67379b653a5de12e7cb687e6eba89711041144bc021f415b43f8b16368"}
Oct 30 14:47:36.993861 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:36.993713    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:37.229615 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:37.229579    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:37.850076 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:37.850048    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-grqw8" event={"ID":"82e0ad12-29e4-4892-841b-801edb30b567","Type":"ContainerStarted","Data":"5f30a480f8da2ce03adf1ef88c12788e48d1c7b7c73561bb35dbfa3d40c70623"}
Oct 30 14:47:37.850484 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:37.850104    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-grqw8" event={"ID":"82e0ad12-29e4-4892-841b-801edb30b567","Type":"ContainerStarted","Data":"8878504e39771303afb119b32cea758c3b248508623a0a222ede2c0f1da68af6"}
Oct 30 14:47:38.856415 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:38.856383    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fr4t" event={"ID":"bef2ba8e-37ad-41c2-a322-b481ba3be1e1","Type":"ContainerStarted","Data":"3442eb4139604a21460a6c2de9a3129477c3c97e2866b64c31b022a7e05480b8"}
Oct 30 14:47:38.856864 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:38.856442    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fr4t" event={"ID":"bef2ba8e-37ad-41c2-a322-b481ba3be1e1","Type":"ContainerStarted","Data":"2230347c8e35f2cf1b17e156afce6b431e32ad025be8775cafb24f0481ed02ee"}
Oct 30 14:47:38.856864 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:38.856470    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:38.858077 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:38.858051    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dtg77" event={"ID":"d375dcba-a3e9-40ec-9e41-fb7e692430f4","Type":"ContainerStarted","Data":"001e359b8040b5caf8d9d1fa937e1adff5c60fa03029d61763dce8e636f84810"}
Oct 30 14:47:39.542223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:39.542181    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:39.544606 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:39.544580    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f43cc708-9756-48e8-a788-a79a9a016e9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n86\" (UID: \"f43cc708-9756-48e8-a788-a79a9a016e9a\") " pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:39.843806 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:39.843709    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4n86"
Oct 30 14:47:40.172069 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:40.172039    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n86"]
Oct 30 14:47:40.175890 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:40.175857    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43cc708_9756_48e8_a788_a79a9a016e9a.slice/crio-79ded5d030ccf7c2ca055f6c25d1b96542aabd14f1bd2502c91fea298f9aef16 WatchSource:0}: Error finding container 79ded5d030ccf7c2ca055f6c25d1b96542aabd14f1bd2502c91fea298f9aef16: Status 404 returned error can't find the container with id 79ded5d030ccf7c2ca055f6c25d1b96542aabd14f1bd2502c91fea298f9aef16
Oct 30 14:47:40.869558 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:40.869519    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n86" event={"ID":"f43cc708-9756-48e8-a788-a79a9a016e9a","Type":"ContainerStarted","Data":"f64eecc2d13c47a3ed2b143b841e652a6e486570d06860d4f99ca881320aa00b"}
Oct 30 14:47:40.869715 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:40.869593    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n86" event={"ID":"f43cc708-9756-48e8-a788-a79a9a016e9a","Type":"ContainerStarted","Data":"79ded5d030ccf7c2ca055f6c25d1b96542aabd14f1bd2502c91fea298f9aef16"}
Oct 30 14:47:41.876448 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:41.876412    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n86" event={"ID":"f43cc708-9756-48e8-a788-a79a9a016e9a","Type":"ContainerStarted","Data":"54a6b37db602423bc113d75001db664e256f935feece106b4cfb2f4c0b1442f1"}
Oct 30 14:47:42.882198 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:42.882158    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n86" event={"ID":"f43cc708-9756-48e8-a788-a79a9a016e9a","Type":"ContainerStarted","Data":"a1dc5d48caf26f63616adb60983bfc3d39cd791fa4d6a98493ceb1cfc55cac86"}
Oct 30 14:47:46.745080 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:46.745047    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4c5w"
Oct 30 14:47:47.235234 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:47.235184    5064 patch_prober.go:29] interesting pod/image-registry-b65c48686-qftvr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 14:47:47.235414 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:47.235252    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b65c48686-qftvr" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:47:48.865423 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:48.865388    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2fr4t"
Oct 30 14:47:49.242587 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.242553    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:47:49.245520 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.245492    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.287707 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.287668    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:47:49.288107 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.288089    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"]
Oct 30 14:47:49.290609 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.290583    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:49.300868 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.300803    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:47:49.305468 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.305443    5064 patch_prober.go:29] interesting pod/image-registry-766565bc5f-b7l2r container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 14:47:49.305587 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.305489    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" podUID="cbee651d-49b7-4013-b903-da36898913cf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:47:49.309176 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.309151    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:47:49.311633 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.311613    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mgcvn"]
Oct 30 14:47:49.311748 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.311732    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.313885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.313866    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.322789 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.322761    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6786q\" (UniqueName: \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.322915 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.322806    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.322915 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.322895    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.323012 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.322984    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f83a575-78ea-4cc2-8bec-82f21f20beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v6pj9\" (UID: \"7f83a575-78ea-4cc2-8bec-82f21f20beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:49.323083 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.323067    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.335144 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.335118    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"]
Oct 30 14:47:49.342546 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.342517    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jmjml\""
Oct 30 14:47:49.349722 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.349698    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 14:47:49.349722 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.349701    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 14:47:49.349940 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.349735    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 14:47:49.349940 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.349769    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 14:47:49.349940 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.349792    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 14:47:49.356428 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.356407    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 14:47:49.362520 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.362495    5064 patch_prober.go:29] interesting pod/image-registry-b65c48686-qftvr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 14:47:49.362625 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.362539    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b65c48686-qftvr" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:47:49.366883 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.366859    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:47:49.366979 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.366921    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mgcvn"]
Oct 30 14:47:49.424518 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424479    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.424518 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424520    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.424772 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424542    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc04fbc7-880f-411b-b2bd-0e59c0368d54-tmp\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.424772 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424564    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.424772 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424716    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.424993 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424781    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6gf\" (UniqueName: \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.424993 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424816    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6786q\" (UniqueName: \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.424993 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424919    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.424993 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424950    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.424996    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425023    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425052    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425060    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f83a575-78ea-4cc2-8bec-82f21f20beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v6pj9\" (UID: \"7f83a575-78ea-4cc2-8bec-82f21f20beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425086    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f725\" (UniqueName: \"kubernetes.io/projected/bc04fbc7-880f-411b-b2bd-0e59c0368d54-kube-api-access-7f725\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.425188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425135    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.425706 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.425681    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.426033 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.426016    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.427594 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.427578    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f83a575-78ea-4cc2-8bec-82f21f20beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v6pj9\" (UID: \"7f83a575-78ea-4cc2-8bec-82f21f20beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:49.435399 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.435378    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd64b66d-fn2zw"]
Oct 30 14:47:49.437749 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.437734    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.441559 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.441518    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 14:47:49.441663 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.441597    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 14:47:49.441729 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.441707    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-mclc7\""
Oct 30 14:47:49.441792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.441777    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ng4tk\""
Oct 30 14:47:49.455547 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.455524    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6786q\" (UniqueName: \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\") pod \"periodic-gathering-lbw9l-kd8m9\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") " pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.461178 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.461160    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kd44p"
Oct 30 14:47:49.466170 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.465440    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd64b66d-fn2zw"]
Oct 30 14:47:49.525804 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.525718    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05be3936-5126-4d01-ba32-06cb852138da-ca-trust-extracted\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.525804 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.525759    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.525804 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.525787    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-registry-tls\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526099 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.525873    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-trusted-ca\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526099 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526025    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526099 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526065    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f725\" (UniqueName: \"kubernetes.io/projected/bc04fbc7-880f-411b-b2bd-0e59c0368d54-kube-api-access-7f725\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.526255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526131    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526163    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-installation-pull-secrets\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526234    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526476 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526346    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-bound-sa-token\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526491    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc04fbc7-880f-411b-b2bd-0e59c0368d54-tmp\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.526540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526535    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526594    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-registry-certificates\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526634    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526764 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526705    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-image-registry-private-configuration\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526822 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526797    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6gf\" (UniqueName: \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.526912 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526854    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78k5\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-kube-api-access-b78k5\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.526991 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526965    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc04fbc7-880f-411b-b2bd-0e59c0368d54-tmp\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.527047 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.526984    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.527199 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.527178    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.527401 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.527382    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.528625 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.528604    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.528926 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.528910    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.558158 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.558132    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:47:49.569718 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.569688    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6gf\" (UniqueName: \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\") pod \"console-8567955586-z5b8w\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") " pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.570798 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.570770    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f725\" (UniqueName: \"kubernetes.io/projected/bc04fbc7-880f-411b-b2bd-0e59c0368d54-kube-api-access-7f725\") pod \"downloads-6dd789c8b7-mgcvn\" (UID: \"bc04fbc7-880f-411b-b2bd-0e59c0368d54\") " pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.604527 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.604490    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:49.625261 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.625227    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627302    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-installation-pull-secrets\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627335    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-bound-sa-token\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627368    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-registry-certificates\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627398    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-image-registry-private-configuration\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627420    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78k5\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-kube-api-access-b78k5\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627439    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05be3936-5126-4d01-ba32-06cb852138da-ca-trust-extracted\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627457    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-registry-tls\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.627471    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-trusted-ca\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.628526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.628174    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-trusted-ca\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.629846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.629812    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:47:49.629986 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.629968    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05be3936-5126-4d01-ba32-06cb852138da-registry-certificates\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.630711 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.630213    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05be3936-5126-4d01-ba32-06cb852138da-ca-trust-extracted\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.631672 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.631614    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-installation-pull-secrets\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.632903 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.632857    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05be3936-5126-4d01-ba32-06cb852138da-image-registry-private-configuration\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.633464 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.633447    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-registry-tls\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.657039 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.657006    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78k5\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-kube-api-access-b78k5\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.658694 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.658665    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05be3936-5126-4d01-ba32-06cb852138da-bound-sa-token\") pod \"image-registry-6dd64b66d-fn2zw\" (UID: \"05be3936-5126-4d01-ba32-06cb852138da\") " pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.748947 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.748891    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:49.858733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.858601    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:47:49.861408 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:49.861382    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd96d74b_8ffc_4011_be71_d831e516f3b6.slice/crio-62b8adfa568681bc22fd53cb47fca914751f053d741d61d693b134bb517482db WatchSource:0}: Error finding container 62b8adfa568681bc22fd53cb47fca914751f053d741d61d693b134bb517482db: Status 404 returned error can't find the container with id 62b8adfa568681bc22fd53cb47fca914751f053d741d61d693b134bb517482db
Oct 30 14:47:49.911951 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.911909    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" event={"ID":"fd96d74b-8ffc-4011-be71-d831e516f3b6","Type":"ContainerStarted","Data":"62b8adfa568681bc22fd53cb47fca914751f053d741d61d693b134bb517482db"}
Oct 30 14:47:49.916737 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.916713    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"]
Oct 30 14:47:49.920406 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:49.920368    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f83a575_78ea_4cc2_8bec_82f21f20beb9.slice/crio-8e25e6a58c9c648d52b3e51e6e5c7997957dc3bc32c6c73055325875169b4065 WatchSource:0}: Error finding container 8e25e6a58c9c648d52b3e51e6e5c7997957dc3bc32c6c73055325875169b4065: Status 404 returned error can't find the container with id 8e25e6a58c9c648d52b3e51e6e5c7997957dc3bc32c6c73055325875169b4065
Oct 30 14:47:49.962872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.962821    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:47:49.966059 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:49.966020    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d92f2f4_6167_45df_b575_084a15332e52.slice/crio-c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085 WatchSource:0}: Error finding container c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085: Status 404 returned error can't find the container with id c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085
Oct 30 14:47:49.978011 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:49.977925    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mgcvn"]
Oct 30 14:47:49.980856 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:49.980817    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc04fbc7_880f_411b_b2bd_0e59c0368d54.slice/crio-5aa32a2a5ce54cca6cc06c62dee86c9451bd0a63b8f29f66a74d74c7414afbf3 WatchSource:0}: Error finding container 5aa32a2a5ce54cca6cc06c62dee86c9451bd0a63b8f29f66a74d74c7414afbf3: Status 404 returned error can't find the container with id 5aa32a2a5ce54cca6cc06c62dee86c9451bd0a63b8f29f66a74d74c7414afbf3
Oct 30 14:47:50.054792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.054707    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd64b66d-fn2zw"]
Oct 30 14:47:50.058191 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:50.058162    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05be3936_5126_4d01_ba32_06cb852138da.slice/crio-1c104ede7915f71fe8c21defd9650b70a20feca7354bb2ad6d496f15974d551f WatchSource:0}: Error finding container 1c104ede7915f71fe8c21defd9650b70a20feca7354bb2ad6d496f15974d551f: Status 404 returned error can't find the container with id 1c104ede7915f71fe8c21defd9650b70a20feca7354bb2ad6d496f15974d551f
Oct 30 14:47:50.938441 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.938403    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" event={"ID":"fd96d74b-8ffc-4011-be71-d831e516f3b6","Type":"ContainerStarted","Data":"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"}
Oct 30 14:47:50.940131 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.940081    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mgcvn" event={"ID":"bc04fbc7-880f-411b-b2bd-0e59c0368d54","Type":"ContainerStarted","Data":"5aa32a2a5ce54cca6cc06c62dee86c9451bd0a63b8f29f66a74d74c7414afbf3"}
Oct 30 14:47:50.941700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.941666    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567955586-z5b8w" event={"ID":"8d92f2f4-6167-45df-b575-084a15332e52","Type":"ContainerStarted","Data":"c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085"}
Oct 30 14:47:50.943392 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.943328    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9" event={"ID":"7f83a575-78ea-4cc2-8bec-82f21f20beb9","Type":"ContainerStarted","Data":"8e25e6a58c9c648d52b3e51e6e5c7997957dc3bc32c6c73055325875169b4065"}
Oct 30 14:47:50.945519 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.945488    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw" event={"ID":"05be3936-5126-4d01-ba32-06cb852138da","Type":"ContainerStarted","Data":"1e0d44b0185ccd798f4c2e7906b142f639611896169b2a3a5fdd526daf0e7f6c"}
Oct 30 14:47:50.945628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.945525    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw" event={"ID":"05be3936-5126-4d01-ba32-06cb852138da","Type":"ContainerStarted","Data":"1c104ede7915f71fe8c21defd9650b70a20feca7354bb2ad6d496f15974d551f"}
Oct 30 14:47:50.945689 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.945627    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:47:50.984478 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:50.984403    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" podStartSLOduration=1.9843839239999999 podStartE2EDuration="1.984383924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:50.983166365 +0000 UTC m=+20.425057083" watchObservedRunningTime="2026-10-30 14:47:50.984383924 +0000 UTC m=+20.426274642"
Oct 30 14:47:51.033126 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.032592    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw" podStartSLOduration=2.032576166 podStartE2EDuration="2.032576166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:51.031210869 +0000 UTC m=+20.473101599" watchObservedRunningTime="2026-10-30 14:47:51.032576166 +0000 UTC m=+20.474466877"
Oct 30 14:47:51.298649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.298577    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fr4t_bef2ba8e-37ad-41c2-a322-b481ba3be1e1/dns/0.log"
Oct 30 14:47:51.499290 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.499255    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fr4t_bef2ba8e-37ad-41c2-a322-b481ba3be1e1/kube-rbac-proxy/0.log"
Oct 30 14:47:51.953939 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.953899    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9" event={"ID":"7f83a575-78ea-4cc2-8bec-82f21f20beb9","Type":"ContainerStarted","Data":"18ab429e4ff6ec896d07c3a961661c5ab0f475254490c1a34ba273c9495c4875"}
Oct 30 14:47:51.954408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.954109    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:51.959944 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.959916    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9"
Oct 30 14:47:51.998163 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:51.997152    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v6pj9" podStartSLOduration=1.8842552750000001 podStartE2EDuration="2.997132711s" totalImagesPullingTime="1.112877436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:51.996308757 +0000 UTC m=+21.438199474" watchObservedRunningTime="2026-10-30 14:47:51.997132711 +0000 UTC m=+21.439023425"
Oct 30 14:47:52.910317 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:52.910285    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vnzrb_43d22383-d225-4613-baf3-30a0b288519e/dns-node-resolver/0.log"
Oct 30 14:47:53.101922 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:53.101875    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dd64b66d-fn2zw_05be3936-5126-4d01-ba32-06cb852138da/registry/0.log"
Oct 30 14:47:53.301372 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:53.301289    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-766565bc5f-b7l2r_cbee651d-49b7-4013-b903-da36898913cf/registry/0.log"
Oct 30 14:47:53.499716 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:53.499687    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b65c48686-qftvr_8407bb01-c8e0-4624-bd80-c27459b6c96e/registry/0.log"
Oct 30 14:47:53.698726 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:53.698698    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96lz7_9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/node-ca/0.log"
Oct 30 14:47:53.965906 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:53.965867    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567955586-z5b8w" event={"ID":"8d92f2f4-6167-45df-b575-084a15332e52","Type":"ContainerStarted","Data":"3f416c9d19632b75f6c681efdc7548bd2369ec292c9675d7cebff9fef8405718"}
Oct 30 14:47:54.011749 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:54.011688    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8567955586-z5b8w" podStartSLOduration=1.464277734 podStartE2EDuration="5.011672235s" totalImagesPullingTime="3.547394501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:54.010863134 +0000 UTC m=+23.452753851" watchObservedRunningTime="2026-10-30 14:47:54.011672235 +0000 UTC m=+23.453562952"
Oct 30 14:47:54.299811 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:54.299702    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58ddf469d5-lvlsc_9e34f75c-8c90-48eb-b5e6-4d363017a5f3/router/0.log"
Oct 30 14:47:54.499088 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:54.499050    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9849b_b7f9e692-eeb3-4000-80dc-0eebeb13f1db/serve-healthcheck-canary/0.log"
Oct 30 14:47:57.236897 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:57.236866    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:47:58.012853 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.010991    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vgxtr"]
Oct 30 14:47:58.036506 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.036475    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.054159 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.054052    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 14:47:58.054159 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.054103    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 14:47:58.054159 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.054136    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mvkp\""
Oct 30 14:47:58.067523 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.067476    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 14:47:58.067664 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.067528    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 14:47:58.114704 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114659    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.114888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114729    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-root\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.114888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114748    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphdl\" (UniqueName: \"kubernetes.io/projected/97b28f19-4139-4144-98af-3a0a302b2e16-kube-api-access-qphdl\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.114888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114818    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-tls\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.114888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114877    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-textfile\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.115089 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.114908    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-metrics-client-ca\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.115089 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.115011    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-wtmp\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.115089 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.115056    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.115189 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.115091    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-sys\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215777 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215741    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-root\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215783    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphdl\" (UniqueName: \"kubernetes.io/projected/97b28f19-4139-4144-98af-3a0a302b2e16-kube-api-access-qphdl\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215815    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-tls\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215851    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-textfile\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215872    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-metrics-client-ca\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215949    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-wtmp\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.215973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.215972    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216275 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216004    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-sys\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216275 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216032    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216628    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-wtmp\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216771 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216686    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216771 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216695    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-sys\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216771 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216759    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-textfile\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216808    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b28f19-4139-4144-98af-3a0a302b2e16-metrics-client-ca\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.216920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.216812    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97b28f19-4139-4144-98af-3a0a302b2e16-root\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.219067 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.219044    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-tls\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.219243 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.219221    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97b28f19-4139-4144-98af-3a0a302b2e16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.240879 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.240847    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphdl\" (UniqueName: \"kubernetes.io/projected/97b28f19-4139-4144-98af-3a0a302b2e16-kube-api-access-qphdl\") pod \"node-exporter-vgxtr\" (UID: \"97b28f19-4139-4144-98af-3a0a302b2e16\") " pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.411355 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.411285    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgxtr"
Oct 30 14:47:58.448163 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:47:58.448127    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b28f19_4139_4144_98af_3a0a302b2e16.slice/crio-233186bee4727bd83c74697a9c5a278eddbd3cb5fff0a9efed3e74cdcfa142ec WatchSource:0}: Error finding container 233186bee4727bd83c74697a9c5a278eddbd3cb5fff0a9efed3e74cdcfa142ec: Status 404 returned error can't find the container with id 233186bee4727bd83c74697a9c5a278eddbd3cb5fff0a9efed3e74cdcfa142ec
Oct 30 14:47:58.992235 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:58.992139    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgxtr" event={"ID":"97b28f19-4139-4144-98af-3a0a302b2e16","Type":"ContainerStarted","Data":"233186bee4727bd83c74697a9c5a278eddbd3cb5fff0a9efed3e74cdcfa142ec"}
Oct 30 14:47:59.307797 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:59.307482    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:47:59.626506 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:59.626460    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:59.626678 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:59.626533    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:59.632431 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:47:59.632402    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:47:59.637753 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:47:59.637718    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:48:00.000701 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:00.000676    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:48:01.002379 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:01.002340    5064 generic.go:356] "Generic (PLEG): container finished" podID="97b28f19-4139-4144-98af-3a0a302b2e16" containerID="ecd2da5186be54a3454dccd3326d136c988fe460ba567e728ca4c1ef54941c0a" exitCode=0
Oct 30 14:48:01.002790 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:01.002402    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgxtr" event={"ID":"97b28f19-4139-4144-98af-3a0a302b2e16","Type":"ContainerDied","Data":"ecd2da5186be54a3454dccd3326d136c988fe460ba567e728ca4c1ef54941c0a"}
Oct 30 14:48:02.010435 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:02.010385    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgxtr" event={"ID":"97b28f19-4139-4144-98af-3a0a302b2e16","Type":"ContainerStarted","Data":"9bf61a53da89b51ab5a9ef0f298ca9b13b7945fe2740ddbd55f98a94cebd7588"}
Oct 30 14:48:02.010863 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:02.010503    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgxtr" event={"ID":"97b28f19-4139-4144-98af-3a0a302b2e16","Type":"ContainerStarted","Data":"22a2ba6ff2c9e1e14b7fe0ae8a9d73aee9b34c039fdc25f619585138a270cd4b"}
Oct 30 14:48:02.070746 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:02.070691    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vgxtr" podStartSLOduration=3.530191833 podStartE2EDuration="5.070673441s" totalImagesPullingTime="1.444363192s" totalInitContainerRuntime="96.118416ms" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:02.069257762 +0000 UTC m=+31.511148479" watchObservedRunningTime="2026-10-30 14:48:02.070673441 +0000 UTC m=+31.512564154"
Oct 30 14:48:04.412412 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:04.412382    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zxt6d"
Oct 30 14:48:07.047508 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:07.047463    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mgcvn" event={"ID":"bc04fbc7-880f-411b-b2bd-0e59c0368d54","Type":"ContainerStarted","Data":"7974b90a7a151e5a309fa3088b63707cd82985e3939d460a0b0cac41dcf69c4a"}
Oct 30 14:48:07.048053 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:07.047709    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:48:07.068474 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:07.068435    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mgcvn"
Oct 30 14:48:07.121110 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:07.121048    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mgcvn" podStartSLOduration=1.8045035550000001 podStartE2EDuration="18.121022023s" totalImagesPullingTime="16.316518468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:07.106789055 +0000 UTC m=+36.548679772" watchObservedRunningTime="2026-10-30 14:48:07.121022023 +0000 UTC m=+36.562912743"
Oct 30 14:48:11.969092 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:11.968284    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd64b66d-fn2zw"
Oct 30 14:48:12.065425 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:12.065391    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:48:14.326601 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.326555    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" podUID="cbee651d-49b7-4013-b903-da36898913cf" containerName="registry" containerID="cri-o://ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d" gracePeriod=30
Oct 30 14:48:14.627405 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.627368    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:48:14.794399 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794362    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794418    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794459    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794497    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794538    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794566    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794590    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.794631 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.794613    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") pod \"cbee651d-49b7-4013-b903-da36898913cf\" (UID: \"cbee651d-49b7-4013-b903-da36898913cf\") "
Oct 30 14:48:14.795229 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.795107    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:14.795553 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.795524    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:14.795856 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.795816    5064 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-registry-certificates\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.795947 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.795863    5064 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbee651d-49b7-4013-b903-da36898913cf-trusted-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.797271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.797245    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:14.797390 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.797261    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:14.797390 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.797325    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:14.797704 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.797685    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "kube-api-access-rdw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:14.797845 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.797810    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:14.801843 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.801805    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted" pod "cbee651d-49b7-4013-b903-da36898913cf" (UID: "cbee651d-49b7-4013-b903-da36898913cf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:14.896915 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896882    5064 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbee651d-49b7-4013-b903-da36898913cf-ca-trust-extracted\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.896915 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896913    5064 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-image-registry-private-configuration\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.897140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896924    5064 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-registry-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.897140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896934    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdw76\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-kube-api-access-rdw76\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.897140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896943    5064 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbee651d-49b7-4013-b903-da36898913cf-installation-pull-secrets\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:14.897140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:14.896953    5064 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbee651d-49b7-4013-b903-da36898913cf-bound-sa-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:15.102620 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.102584    5064 generic.go:356] "Generic (PLEG): container finished" podID="cbee651d-49b7-4013-b903-da36898913cf" containerID="ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d" exitCode=0
Oct 30 14:48:15.102808 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.102640    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r"
Oct 30 14:48:15.102808 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.102664    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" event={"ID":"cbee651d-49b7-4013-b903-da36898913cf","Type":"ContainerDied","Data":"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"}
Oct 30 14:48:15.102808 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.102722    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766565bc5f-b7l2r" event={"ID":"cbee651d-49b7-4013-b903-da36898913cf","Type":"ContainerDied","Data":"439bf64a229cbc532e15768f15370156202cc95f330ca494ada3639c41921184"}
Oct 30 14:48:15.102808 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.102744    5064 scope.go:122] "RemoveContainer" containerID="ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"
Oct 30 14:48:15.112255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.112237    5064 scope.go:122] "RemoveContainer" containerID="ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"
Oct 30 14:48:15.112551 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:15.112531    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d\": container with ID starting with ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d not found: ID does not exist" containerID="ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"
Oct 30 14:48:15.112598 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.112560    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d"} err="failed to get container status \"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d\": rpc error: code = NotFound desc = could not find container \"ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d\": container with ID starting with ecfdbacb65879eea56602d9ffc445afbbe3b719690a02a1b2fb59292908b0c0d not found: ID does not exist"
Oct 30 14:48:15.160370 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.160272    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:48:15.172754 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:15.172720    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-766565bc5f-b7l2r"]
Oct 30 14:48:16.721126 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:16.721089    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbee651d-49b7-4013-b903-da36898913cf" path="/var/lib/kubelet/pods/cbee651d-49b7-4013-b903-da36898913cf/volumes"
Oct 30 14:48:17.963219 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:17.963178    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:48:22.524385 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.524343    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:48:22.524938 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.524586    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" podUID="bad3a326-d346-4956-b8ef-d29ca51b43a1" containerName="insights-gathering" containerID="cri-o://33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5" gracePeriod=30
Oct 30 14:48:22.864087 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.864057    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:48:22.969138 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969105    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") pod \"bad3a326-d346-4956-b8ef-d29ca51b43a1\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") "
Oct 30 14:48:22.969320 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969191    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") pod \"bad3a326-d346-4956-b8ef-d29ca51b43a1\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") "
Oct 30 14:48:22.969320 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969291    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") pod \"bad3a326-d346-4956-b8ef-d29ca51b43a1\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") "
Oct 30 14:48:22.969320 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969317    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") pod \"bad3a326-d346-4956-b8ef-d29ca51b43a1\" (UID: \"bad3a326-d346-4956-b8ef-d29ca51b43a1\") "
Oct 30 14:48:22.969460 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969437    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path" pod "bad3a326-d346-4956-b8ef-d29ca51b43a1" (UID: "bad3a326-d346-4956-b8ef-d29ca51b43a1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:22.969518 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969463    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle" pod "bad3a326-d346-4956-b8ef-d29ca51b43a1" (UID: "bad3a326-d346-4956-b8ef-d29ca51b43a1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:22.969606 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969585    5064 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bad3a326-d346-4956-b8ef-d29ca51b43a1-archives-path\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:22.969658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969608    5064 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-service-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:22.969658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.969643    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle" pod "bad3a326-d346-4956-b8ef-d29ca51b43a1" (UID: "bad3a326-d346-4956-b8ef-d29ca51b43a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:22.971650 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:22.971622    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg" pod "bad3a326-d346-4956-b8ef-d29ca51b43a1" (UID: "bad3a326-d346-4956-b8ef-d29ca51b43a1"). InnerVolumeSpecName "kube-api-access-mvbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:23.070891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.070849    5064 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3a326-d346-4956-b8ef-d29ca51b43a1-trusted-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:23.070891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.070888    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbhg\" (UniqueName: \"kubernetes.io/projected/bad3a326-d346-4956-b8ef-d29ca51b43a1-kube-api-access-mvbhg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:23.138666 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.138627    5064 generic.go:356] "Generic (PLEG): container finished" podID="bad3a326-d346-4956-b8ef-d29ca51b43a1" containerID="33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5" exitCode=2
Oct 30 14:48:23.138846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.138674    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2"
Oct 30 14:48:23.138846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.138712    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" event={"ID":"bad3a326-d346-4956-b8ef-d29ca51b43a1","Type":"ContainerDied","Data":"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"}
Oct 30 14:48:23.138846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.138765    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dzxvg-c8zl2" event={"ID":"bad3a326-d346-4956-b8ef-d29ca51b43a1","Type":"ContainerDied","Data":"512b71d6ec5b87baf2e0b56e568a6a01787f0fdb142ff001f813f1f9b5ba8eb2"}
Oct 30 14:48:23.138846 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.138785    5064 scope.go:122] "RemoveContainer" containerID="33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"
Oct 30 14:48:23.147797 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.147780    5064 scope.go:122] "RemoveContainer" containerID="33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"
Oct 30 14:48:23.148127 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:23.148106    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5\": container with ID starting with 33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5 not found: ID does not exist" containerID="33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"
Oct 30 14:48:23.148188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.148142    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5"} err="failed to get container status \"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5\": rpc error: code = NotFound desc = could not find container \"33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5\": container with ID starting with 33b7e757e54b3515d9e05fbfd2970e056d4156c3c419de99a1bcf86ecf1477a5 not found: ID does not exist"
Oct 30 14:48:23.193555 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.193524    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:48:23.207205 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:23.207174    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-dzxvg-c8zl2"]
Oct 30 14:48:24.721161 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:24.721124    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad3a326-d346-4956-b8ef-d29ca51b43a1" path="/var/lib/kubelet/pods/bad3a326-d346-4956-b8ef-d29ca51b43a1/volumes"
Oct 30 14:48:29.627726 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:29.627682    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:48:37.092098 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.092035    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b65c48686-qftvr" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry" containerID="cri-o://5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898" gracePeriod=30
Oct 30 14:48:37.231353 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.231321    5064 patch_prober.go:29] interesting pod/image-registry-b65c48686-qftvr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.18:5000/healthz\": dial tcp 10.132.0.18:5000: connect: connection refused" start-of-body=
Oct 30 14:48:37.231481 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.231392    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b65c48686-qftvr" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.18:5000/healthz\": dial tcp 10.132.0.18:5000: connect: connection refused"
Oct 30 14:48:37.378877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.378847    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:48:37.472041 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472010    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472041 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472046    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472087    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472111    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472147    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472173    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472203    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.472271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472252    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") pod \"8407bb01-c8e0-4624-bd80-c27459b6c96e\" (UID: \"8407bb01-c8e0-4624-bd80-c27459b6c96e\") "
Oct 30 14:48:37.473109 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472739    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:37.473109 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.472848    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:37.474583 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.474559    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:37.474792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.474753    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:37.474949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.474822    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:37.475014 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.474964    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "kube-api-access-sv4fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:37.475078 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.475061    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:37.479697 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.479671    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted" pod "8407bb01-c8e0-4624-bd80-c27459b6c96e" (UID: "8407bb01-c8e0-4624-bd80-c27459b6c96e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:37.573480 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573439    5064 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-bound-sa-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573480 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573475    5064 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-certificates\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573488    5064 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-image-registry-private-configuration\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573499    5064 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8407bb01-c8e0-4624-bd80-c27459b6c96e-ca-trust-extracted\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573508    5064 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8407bb01-c8e0-4624-bd80-c27459b6c96e-installation-pull-secrets\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573517    5064 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8407bb01-c8e0-4624-bd80-c27459b6c96e-trusted-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573525    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4fm\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-kube-api-access-sv4fm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.573671 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:37.573534    5064 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8407bb01-c8e0-4624-bd80-c27459b6c96e-registry-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:38.208075 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.208035    5064 generic.go:356] "Generic (PLEG): container finished" podID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerID="5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898" exitCode=0
Oct 30 14:48:38.208491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.208086    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b65c48686-qftvr"
Oct 30 14:48:38.208491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.208122    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b65c48686-qftvr" event={"ID":"8407bb01-c8e0-4624-bd80-c27459b6c96e","Type":"ContainerDied","Data":"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"}
Oct 30 14:48:38.208491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.208181    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b65c48686-qftvr" event={"ID":"8407bb01-c8e0-4624-bd80-c27459b6c96e","Type":"ContainerDied","Data":"8cc5c7a1f2d3deafca3b6f5a68db6a1602e20428fb5f9dba979e5ffc17557ed6"}
Oct 30 14:48:38.208491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.208204    5064 scope.go:122] "RemoveContainer" containerID="5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"
Oct 30 14:48:38.217054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.217036    5064 scope.go:122] "RemoveContainer" containerID="5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"
Oct 30 14:48:38.217334 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:38.217313    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898\": container with ID starting with 5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898 not found: ID does not exist" containerID="5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"
Oct 30 14:48:38.217404 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.217341    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898"} err="failed to get container status \"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898\": rpc error: code = NotFound desc = could not find container \"5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898\": container with ID starting with 5df7aa01afee3c227f542819a443e085a5196bb7f8137d919c27cd1fb0ec1898 not found: ID does not exist"
Oct 30 14:48:38.277769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.277738    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:48:38.291448 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.291423    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b65c48686-qftvr"]
Oct 30 14:48:38.720217 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:38.720176    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" path="/var/lib/kubelet/pods/8407bb01-c8e0-4624-bd80-c27459b6c96e/volumes"
Oct 30 14:48:40.219369 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.219334    5064 generic.go:356] "Generic (PLEG): container finished" podID="09ec3816-9582-4e08-b9d0-9d03174f5628" containerID="e014cf88bfd9b477e3889e3288fa4d67af49ee817b82da43369fc76393f88a94" exitCode=0
Oct 30 14:48:40.219794 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.219404    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm" event={"ID":"09ec3816-9582-4e08-b9d0-9d03174f5628","Type":"ContainerDied","Data":"e014cf88bfd9b477e3889e3288fa4d67af49ee817b82da43369fc76393f88a94"}
Oct 30 14:48:40.219794 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.219775    5064 scope.go:122] "RemoveContainer" containerID="e014cf88bfd9b477e3889e3288fa4d67af49ee817b82da43369fc76393f88a94"
Oct 30 14:48:40.221052 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.221031    5064 generic.go:356] "Generic (PLEG): container finished" podID="b767b7e7-8048-4086-aa1b-5ec54e3d0568" containerID="e6a342fc6f67b0f1c5f4050e1770f9b49e98db48674797f3e599f1c27314ff90" exitCode=0
Oct 30 14:48:40.221153 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.221102    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg" event={"ID":"b767b7e7-8048-4086-aa1b-5ec54e3d0568","Type":"ContainerDied","Data":"e6a342fc6f67b0f1c5f4050e1770f9b49e98db48674797f3e599f1c27314ff90"}
Oct 30 14:48:40.221391 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:40.221378    5064 scope.go:122] "RemoveContainer" containerID="e6a342fc6f67b0f1c5f4050e1770f9b49e98db48674797f3e599f1c27314ff90"
Oct 30 14:48:41.227026 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:41.226983    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s5djm" event={"ID":"09ec3816-9582-4e08-b9d0-9d03174f5628","Type":"ContainerStarted","Data":"6ea8ff230d7f81c34b6673dc9498e8f3ec4d241e4ff8c1a45a6211cc12df7688"}
Oct 30 14:48:41.228848 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:41.228809    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tfwwg" event={"ID":"b767b7e7-8048-4086-aa1b-5ec54e3d0568","Type":"ContainerStarted","Data":"f0079ead8ef32c4b970f7979d0925e4a3d16d342a0c4f70a87986a0f2eb0c0bf"}
Oct 30 14:48:42.984609 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:42.984543    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8567955586-z5b8w" podUID="8d92f2f4-6167-45df-b575-084a15332e52" containerName="console" containerID="cri-o://3f416c9d19632b75f6c681efdc7548bd2369ec292c9675d7cebff9fef8405718" gracePeriod=15
Oct 30 14:48:43.239329 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.239285    5064 generic.go:356] "Generic (PLEG): container finished" podID="8d92f2f4-6167-45df-b575-084a15332e52" containerID="3f416c9d19632b75f6c681efdc7548bd2369ec292c9675d7cebff9fef8405718" exitCode=0
Oct 30 14:48:43.239461 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.239357    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567955586-z5b8w" event={"ID":"8d92f2f4-6167-45df-b575-084a15332e52","Type":"ContainerDied","Data":"3f416c9d19632b75f6c681efdc7548bd2369ec292c9675d7cebff9fef8405718"}
Oct 30 14:48:43.239461 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.239425    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567955586-z5b8w" event={"ID":"8d92f2f4-6167-45df-b575-084a15332e52","Type":"ContainerDied","Data":"c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085"}
Oct 30 14:48:43.239461 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.239445    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c66b2e4d18e0ff682618c92bdf497cc89a920db102650ee6b4ec683fe62a7085"
Oct 30 14:48:43.270478 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.270442    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:48:43.325726 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325695    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\" (UniqueName: \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325726 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325733    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325970 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325773    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325970 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325807    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325970 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325857    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325970 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325895    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.325970 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.325933    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\") pod \"8d92f2f4-6167-45df-b575-084a15332e52\" (UID: \"8d92f2f4-6167-45df-b575-084a15332e52\") "
Oct 30 14:48:43.326284 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.326253    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:43.326284 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.326270    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:43.326391 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.326367    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:43.326444 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.326275    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:43.328265 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.328233    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:43.328429 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.328407    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:43.328492 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.328426    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf" pod "8d92f2f4-6167-45df-b575-084a15332e52" (UID: "8d92f2f4-6167-45df-b575-084a15332e52"). InnerVolumeSpecName "kube-api-access-mg6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:43.427207 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427166    5064 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-oauth-config\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427207 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427199    5064 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d92f2f4-6167-45df-b575-084a15332e52-tmp\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427207 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427209    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6gf\" (UniqueName: \"kubernetes.io/projected/8d92f2f4-6167-45df-b575-084a15332e52-kube-api-access-mg6gf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427451 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427220    5064 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d92f2f4-6167-45df-b575-084a15332e52-console-serving-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427451 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427229    5064 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427451 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427238    5064 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-console-config\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:43.427451 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:43.427247    5064 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d92f2f4-6167-45df-b575-084a15332e52-oauth-serving-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:48:44.243089 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:44.243049    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567955586-z5b8w"
Oct 30 14:48:44.299224 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:44.299186    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:48:44.312909 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:44.312880    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8567955586-z5b8w"]
Oct 30 14:48:44.720452 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:44.720415    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d92f2f4-6167-45df-b575-084a15332e52" path="/var/lib/kubelet/pods/8d92f2f4-6167-45df-b575-084a15332e52/volumes"
Oct 30 14:48:56.293505 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:56.293409    5064 generic.go:356] "Generic (PLEG): container finished" podID="107da892-9033-4172-b313-2431ef06d16c" containerID="19ebbae3e427b4be8dcb15461437930a8d9f2255a65fe84d072d58503e616e58" exitCode=0
Oct 30 14:48:56.293505 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:56.293450    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerDied","Data":"19ebbae3e427b4be8dcb15461437930a8d9f2255a65fe84d072d58503e616e58"}
Oct 30 14:48:56.293505 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:56.293504    5064 scope.go:122] "RemoveContainer" containerID="856c8dca650f9f0197a0c3b97999e1cc9ff1541f48a65a77a8cf43985995eaf7"
Oct 30 14:48:56.294088 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:48:56.293856    5064 scope.go:122] "RemoveContainer" containerID="19ebbae3e427b4be8dcb15461437930a8d9f2255a65fe84d072d58503e616e58"
Oct 30 14:48:56.294088 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:56.294069    5064 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-njl5k_openshift-insights(107da892-9033-4172-b313-2431ef06d16c)\"" pod="openshift-insights/insights-operator-7c4986d449-njl5k" podUID="107da892-9033-4172-b313-2431ef06d16c"
Oct 30 14:48:59.628848 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:48:59.628782    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:49:06.720464 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:06.720426    5064 scope.go:122] "RemoveContainer" containerID="19ebbae3e427b4be8dcb15461437930a8d9f2255a65fe84d072d58503e616e58"
Oct 30 14:49:07.270193 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.270155    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:49:07.270436 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.270409    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" podUID="fd96d74b-8ffc-4011-be71-d831e516f3b6" containerName="insights-gathering" containerID="cri-o://43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe" gracePeriod=30
Oct 30 14:49:07.342022 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.341980    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-njl5k" event={"ID":"107da892-9033-4172-b313-2431ef06d16c","Type":"ContainerStarted","Data":"5ebdfe4f1e157ebec073858ae2feb9d5ec4ce1a6e81f952aec8de494e5e02ad9"}
Oct 30 14:49:07.534519 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.534490    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:49:07.628748 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.628714    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\" (UniqueName: \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\") pod \"fd96d74b-8ffc-4011-be71-d831e516f3b6\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") "
Oct 30 14:49:07.628949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.628770    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\") pod \"fd96d74b-8ffc-4011-be71-d831e516f3b6\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") "
Oct 30 14:49:07.628949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.628797    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\") pod \"fd96d74b-8ffc-4011-be71-d831e516f3b6\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") "
Oct 30 14:49:07.628949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.628823    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\") pod \"fd96d74b-8ffc-4011-be71-d831e516f3b6\" (UID: \"fd96d74b-8ffc-4011-be71-d831e516f3b6\") "
Oct 30 14:49:07.629164 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.629138    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path" pod "fd96d74b-8ffc-4011-be71-d831e516f3b6" (UID: "fd96d74b-8ffc-4011-be71-d831e516f3b6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:07.629255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.629228    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle" pod "fd96d74b-8ffc-4011-be71-d831e516f3b6" (UID: "fd96d74b-8ffc-4011-be71-d831e516f3b6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:07.629255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.629241    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle" pod "fd96d74b-8ffc-4011-be71-d831e516f3b6" (UID: "fd96d74b-8ffc-4011-be71-d831e516f3b6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:07.631066 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.631041    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q" pod "fd96d74b-8ffc-4011-be71-d831e516f3b6" (UID: "fd96d74b-8ffc-4011-be71-d831e516f3b6"). InnerVolumeSpecName "kube-api-access-6786q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:07.729559 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.729518    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6786q\" (UniqueName: \"kubernetes.io/projected/fd96d74b-8ffc-4011-be71-d831e516f3b6-kube-api-access-6786q\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:49:07.729559 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.729551    5064 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fd96d74b-8ffc-4011-be71-d831e516f3b6-archives-path\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:49:07.729559 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.729560    5064 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-trusted-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:49:07.729559 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:07.729570    5064 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd96d74b-8ffc-4011-be71-d831e516f3b6-service-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:49:08.347477 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.347438    5064 generic.go:356] "Generic (PLEG): container finished" podID="fd96d74b-8ffc-4011-be71-d831e516f3b6" containerID="43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe" exitCode=2
Oct 30 14:49:08.347657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.347501    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9"
Oct 30 14:49:08.347657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.347529    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" event={"ID":"fd96d74b-8ffc-4011-be71-d831e516f3b6","Type":"ContainerDied","Data":"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"}
Oct 30 14:49:08.347657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.347589    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lbw9l-kd8m9" event={"ID":"fd96d74b-8ffc-4011-be71-d831e516f3b6","Type":"ContainerDied","Data":"62b8adfa568681bc22fd53cb47fca914751f053d741d61d693b134bb517482db"}
Oct 30 14:49:08.347657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.347612    5064 scope.go:122] "RemoveContainer" containerID="43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"
Oct 30 14:49:08.355928 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.355909    5064 scope.go:122] "RemoveContainer" containerID="43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"
Oct 30 14:49:08.356183 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:49:08.356162    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe\": container with ID starting with 43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe not found: ID does not exist" containerID="43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"
Oct 30 14:49:08.356227 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.356194    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe"} err="failed to get container status \"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe\": rpc error: code = NotFound desc = could not find container \"43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe\": container with ID starting with 43567c17c70cf01c1a3fb641d902276326e9e90450bc090730548bf8adc0cdfe not found: ID does not exist"
Oct 30 14:49:08.402445 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.402415    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:49:08.415971 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.415947    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-lbw9l-kd8m9"]
Oct 30 14:49:08.720031 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:49:08.719998    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd96d74b-8ffc-4011-be71-d831e516f3b6" path="/var/lib/kubelet/pods/fd96d74b-8ffc-4011-be71-d831e516f3b6/volumes"
Oct 30 14:49:29.628887 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:49:29.628825    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:49:59.629066 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:49:59.628983    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:50:29.629057 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:50:29.629019    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:50:59.629682 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:50:59.629640    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:51:29.629064 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:51:29.629025    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:51:32.423344 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423303    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"]
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423772    5064 cpu_manager.go:414] "containerMap: removing container" podUID="cbee651d-49b7-4013-b903-da36898913cf" containerName="registry"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423790    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbee651d-49b7-4013-b903-da36898913cf" containerName="registry"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423816    5064 cpu_manager.go:414] "containerMap: removing container" podUID="fd96d74b-8ffc-4011-be71-d831e516f3b6" containerName="insights-gathering"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423823    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd96d74b-8ffc-4011-be71-d831e516f3b6" containerName="insights-gathering"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423852    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8d92f2f4-6167-45df-b575-084a15332e52" containerName="console"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423861    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d92f2f4-6167-45df-b575-084a15332e52" containerName="console"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423874    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry"
Oct 30 14:51:32.423891 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423880    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423910    5064 cpu_manager.go:414] "containerMap: removing container" podUID="bad3a326-d346-4956-b8ef-d29ca51b43a1" containerName="insights-gathering"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.423916    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad3a326-d346-4956-b8ef-d29ca51b43a1" containerName="insights-gathering"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.424002    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad3a326-d346-4956-b8ef-d29ca51b43a1" containerName="insights-gathering"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.424017    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbee651d-49b7-4013-b903-da36898913cf" containerName="registry"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.424029    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd96d74b-8ffc-4011-be71-d831e516f3b6" containerName="insights-gathering"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.424036    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d92f2f4-6167-45df-b575-084a15332e52" containerName="console"
Oct 30 14:51:32.424304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.424046    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8407bb01-c8e0-4624-bd80-c27459b6c96e" containerName="registry"
Oct 30 14:51:32.426448 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.426426    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.474050 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.474023    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 14:51:32.482900 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.482869    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 14:51:32.520381 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520350    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.520551 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520391    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m669m\" (UniqueName: \"kubernetes.io/projected/3897e788-0c71-4f61-8820-11015ce1ac74-kube-api-access-m669m\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.520551 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520418    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.520672 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520582    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.520672 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520614    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.520672 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.520631    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.559171 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.559140    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 14:51:32.559511 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.559497    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 14:51:32.559597 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.559586    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 14:51:32.559651 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.559641    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b8vcv\""
Oct 30 14:51:32.610016 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.609981    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"]
Oct 30 14:51:32.621504 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621472    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.621681 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621508    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.621681 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621604    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.621681 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621633    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.621681 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621665    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m669m\" (UniqueName: \"kubernetes.io/projected/3897e788-0c71-4f61-8820-11015ce1ac74-kube-api-access-m669m\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.621924 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.621701    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.622140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.622116    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.624089 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.624064    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3897e788-0c71-4f61-8820-11015ce1ac74-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.624227 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.624203    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.624334 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.624320    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.624436 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.624414    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3897e788-0c71-4f61-8820-11015ce1ac74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.664614 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.664575    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m669m\" (UniqueName: \"kubernetes.io/projected/3897e788-0c71-4f61-8820-11015ce1ac74-kube-api-access-m669m\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-jknvj\" (UID: \"3897e788-0c71-4f61-8820-11015ce1ac74\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:32.739308 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:32.739217    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:33.154408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:33.154382    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"]
Oct 30 14:51:33.157016 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:51:33.156991    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3897e788_0c71_4f61_8820_11015ce1ac74.slice/crio-5c17448625a598bf3c72057ee4f474b8c4f043fdb9583a5c89ed3bdf7c4d0a85 WatchSource:0}: Error finding container 5c17448625a598bf3c72057ee4f474b8c4f043fdb9583a5c89ed3bdf7c4d0a85: Status 404 returned error can't find the container with id 5c17448625a598bf3c72057ee4f474b8c4f043fdb9583a5c89ed3bdf7c4d0a85
Oct 30 14:51:33.931679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:33.931633    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj" event={"ID":"3897e788-0c71-4f61-8820-11015ce1ac74","Type":"ContainerStarted","Data":"5c17448625a598bf3c72057ee4f474b8c4f043fdb9583a5c89ed3bdf7c4d0a85"}
Oct 30 14:51:36.947803 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:36.947764    5064 generic.go:356] "Generic (PLEG): container finished" podID="3897e788-0c71-4f61-8820-11015ce1ac74" containerID="cb7cbfb15c7915b43ff1b36b08c6177a74e59e6cb5ea58d890297e7276cf339e" exitCode=0
Oct 30 14:51:36.948202 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:36.947847    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj" event={"ID":"3897e788-0c71-4f61-8820-11015ce1ac74","Type":"ContainerDied","Data":"cb7cbfb15c7915b43ff1b36b08c6177a74e59e6cb5ea58d890297e7276cf339e"}
Oct 30 14:51:37.954493 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:37.954454    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj" event={"ID":"3897e788-0c71-4f61-8820-11015ce1ac74","Type":"ContainerStarted","Data":"872c251173b481df16402164ae9bd73411f82ace4fef9a5440fc7cf79bc562d4"}
Oct 30 14:51:37.955000 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:37.954687    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:38.060303 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:38.060234    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj" podStartSLOduration=2.461524035 podStartE2EDuration="6.060217639s" totalImagesPullingTime="3.13665436s" totalInitContainerRuntime="462.039244ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:38.046184533 +0000 UTC m=+247.488075262" watchObservedRunningTime="2026-10-30 14:51:38.060217639 +0000 UTC m=+247.502108354"
Oct 30 14:51:48.960782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:48.960751    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-jknvj"
Oct 30 14:51:53.038243 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.038212    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"]
Oct 30 14:51:53.041304 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.041286    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.073053 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.073009    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 14:51:53.077406 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.077381    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 14:51:53.077773 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.077748    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r5tgf\""
Oct 30 14:51:53.078953 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.078931    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"]
Oct 30 14:51:53.092148 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.092110    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb6r\" (UniqueName: \"kubernetes.io/projected/afabb0f7-990c-4aea-8d89-cfa11b587cff-kube-api-access-vbb6r\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.092297 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.092178    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb0f7-990c-4aea-8d89-cfa11b587cff-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.092297 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.092210    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afabb0f7-990c-4aea-8d89-cfa11b587cff-tmp\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.193114 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.193075    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb0f7-990c-4aea-8d89-cfa11b587cff-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.193283 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.193120    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afabb0f7-990c-4aea-8d89-cfa11b587cff-tmp\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.193283 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.193182    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb6r\" (UniqueName: \"kubernetes.io/projected/afabb0f7-990c-4aea-8d89-cfa11b587cff-kube-api-access-vbb6r\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.195451 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.195420    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afabb0f7-990c-4aea-8d89-cfa11b587cff-tmp\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.195623 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.195602    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb0f7-990c-4aea-8d89-cfa11b587cff-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.222179 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.222143    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb6r\" (UniqueName: \"kubernetes.io/projected/afabb0f7-990c-4aea-8d89-cfa11b587cff-kube-api-access-vbb6r\") pod \"kube-auth-proxy-6d8f97cc56-wk65g\" (UID: \"afabb0f7-990c-4aea-8d89-cfa11b587cff\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.354081 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.353992    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"
Oct 30 14:51:53.659784 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:53.659748    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g"]
Oct 30 14:51:53.662878 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:51:53.662851    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafabb0f7_990c_4aea_8d89_cfa11b587cff.slice/crio-e19dacbd3549a8f87fc44b0c9ea9ec33a2847cd481711455ba30c0b8fc4685f5 WatchSource:0}: Error finding container e19dacbd3549a8f87fc44b0c9ea9ec33a2847cd481711455ba30c0b8fc4685f5: Status 404 returned error can't find the container with id e19dacbd3549a8f87fc44b0c9ea9ec33a2847cd481711455ba30c0b8fc4685f5
Oct 30 14:51:54.026778 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.026691    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g" event={"ID":"afabb0f7-990c-4aea-8d89-cfa11b587cff","Type":"ContainerStarted","Data":"e19dacbd3549a8f87fc44b0c9ea9ec33a2847cd481711455ba30c0b8fc4685f5"}
Oct 30 14:51:54.696693 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.696656    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:51:54.700625 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.700597    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.720626 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.720596    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 14:51:54.720626 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.720620    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 14:51:54.720877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.720670    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 14:51:54.726580 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.726554    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 14:51:54.734222 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.734189    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jz5vp\""
Oct 30 14:51:54.742319 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.742284    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:51:54.807313 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807271    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807313 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807318    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807532 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807389    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807532 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807430    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807532 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807463    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hw2j\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807642 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807558    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.807642 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.807611    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909069 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909031    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909260 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909084    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909260 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909123    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hw2j\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909260 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909150    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909260 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909180    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909260 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909260    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.909553 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.909286    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.910150 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.910110    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.911928 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.911878    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.911928 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.911900    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.912072 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.911902    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.912437 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.912412    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.934877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.934814    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hw2j\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:54.935041 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:54.934905    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8r5f7\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:55.016289 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:55.016211    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:55.336760 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:55.336682    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:51:55.340045 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:51:55.340011    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877d2d9d_98f2_4731_be74_0cc6ef486136.slice/crio-f8b45bb216e6c11d1d2bbd1ce5e7cc74a08dc235cd5dda5ea29f7aa0015ebec9 WatchSource:0}: Error finding container f8b45bb216e6c11d1d2bbd1ce5e7cc74a08dc235cd5dda5ea29f7aa0015ebec9: Status 404 returned error can't find the container with id f8b45bb216e6c11d1d2bbd1ce5e7cc74a08dc235cd5dda5ea29f7aa0015ebec9
Oct 30 14:51:56.038367 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:56.038293    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" event={"ID":"877d2d9d-98f2-4731-be74-0cc6ef486136","Type":"ContainerStarted","Data":"f8b45bb216e6c11d1d2bbd1ce5e7cc74a08dc235cd5dda5ea29f7aa0015ebec9"}
Oct 30 14:51:57.044915 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:57.044866    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g" event={"ID":"afabb0f7-990c-4aea-8d89-cfa11b587cff","Type":"ContainerStarted","Data":"0a78ce999ead1daf1361284a2c709fec4d7d323b9615bfabcd213ccf0e5ace18"}
Oct 30 14:51:57.088782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:57.088728    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-wk65g" podStartSLOduration=2.641950093 podStartE2EDuration="5.088713083s" totalImagesPullingTime="2.44676299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:57.087966663 +0000 UTC m=+266.529857381" watchObservedRunningTime="2026-10-30 14:51:57.088713083 +0000 UTC m=+266.530603797"
Oct 30 14:51:57.812291 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:57.812258    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:57.812359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:57.812326    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:58.051176 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:58.051136    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" event={"ID":"877d2d9d-98f2-4731-be74-0cc6ef486136","Type":"ContainerStarted","Data":"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"}
Oct 30 14:51:58.051607 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:58.051349    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:51:58.053186 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:58.053143    5064 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8r5f7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:51:58.053328 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:58.053217    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:51:58.115006 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:58.114901    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" podStartSLOduration=1.6458739279999999 podStartE2EDuration="4.114885222s" totalImagesPullingTime="2.469011294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:58.114106667 +0000 UTC m=+267.555997384" watchObservedRunningTime="2026-10-30 14:51:58.114885222 +0000 UTC m=+267.556775938"
Oct 30 14:51:59.056255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.056225    5064 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8r5f7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:51:59.056646 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.056276    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:51:59.377203 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.377162    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"]
Oct 30 14:51:59.381112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.381085    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.404533 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.404496    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"]
Oct 30 14:51:59.450181 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.450148    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pf672\""
Oct 30 14:51:59.451406 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451379    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451541 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451412    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5t9c\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-kube-api-access-w5t9c\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451541 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451446    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451541 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451522    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451646 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451627    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451691 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451679    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451789 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451771    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451824 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451807    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.451888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.451865    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.552618 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552568    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.552791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552639    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.552791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552669    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.552791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552707    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.552791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552755    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553042 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552802    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5t9c\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-kube-api-access-w5t9c\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553042 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552859    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553042 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552886    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553042 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.552947    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553245 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.553188    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553407 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.553373    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553536 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.553424    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.553906 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.553879    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.554062 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.554036    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.555364 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.555337    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.555567 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.555549    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.577809 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.577777    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.577965 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.577786    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5t9c\" (UniqueName: \"kubernetes.io/projected/d47b1eb7-c394-4aa6-8969-75a8269a0d02-kube-api-access-w5t9c\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2qv8f\" (UID: \"d47b1eb7-c394-4aa6-8969-75a8269a0d02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:51:59.630014 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:51:59.629920    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:51:59.696901 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:51:59.696861    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:52:00.018345 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:00.016963    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"]
Oct 30 14:52:00.020008 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:52:00.019979    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47b1eb7_c394_4aa6_8969_75a8269a0d02.slice/crio-6510ac89e214563debb572ee55bd17325e18f4e6ace5b33ecac03a919dc8fb45 WatchSource:0}: Error finding container 6510ac89e214563debb572ee55bd17325e18f4e6ace5b33ecac03a919dc8fb45: Status 404 returned error can't find the container with id 6510ac89e214563debb572ee55bd17325e18f4e6ace5b33ecac03a919dc8fb45
Oct 30 14:52:00.061338 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:00.061299    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f" event={"ID":"d47b1eb7-c394-4aa6-8969-75a8269a0d02","Type":"ContainerStarted","Data":"6510ac89e214563debb572ee55bd17325e18f4e6ace5b33ecac03a919dc8fb45"}
Oct 30 14:52:02.057046 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:02.057010    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:52:02.714657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:02.714607    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:02.714795 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:02.714715    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:02.714795 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:02.714757    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:03.078370 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:03.078276    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f" event={"ID":"d47b1eb7-c394-4aa6-8969-75a8269a0d02","Type":"ContainerStarted","Data":"58e9851450ac8160ec03d0f5b0697c698027b1a6dcd3c9bfd6f31af067674a57"}
Oct 30 14:52:03.142002 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:03.141943    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f" podStartSLOduration=1.4503268710000001 podStartE2EDuration="4.14192648s" totalImagesPullingTime="2.691599609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:03.140725937 +0000 UTC m=+272.582616652" watchObservedRunningTime="2026-10-30 14:52:03.14192648 +0000 UTC m=+272.583817196"
Oct 30 14:52:03.697918 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:03.697880    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:52:03.702913 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:03.702887    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:52:04.083628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:04.083545    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:52:04.084508 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:04.084486    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2qv8f"
Oct 30 14:52:29.631007 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:52:29.630963    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:52:30.733862 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:30.733818    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:52:30.735625 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:30.735602    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:52:58.771466 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.771382    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 14:52:58.773903 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.773887    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.805389 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.805355    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 14:52:58.905941 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.905894    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906127 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906014    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcs47\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906127 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906105    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906202 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906141    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906202 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906160    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906202 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906182    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:58.906354 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:58.906336    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007191 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007191 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007191    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007223    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007265    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007294    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007325    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcs47\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.007456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.007393    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.008066 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.008022    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.009742 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.009722    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.009850 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.009777    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.010085 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.010067    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.010149 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.010116    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.032019 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.031947    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcs47\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.032019 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.031970    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vbpm8\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.084400 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.084366    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:52:59.395752 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.395722    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 14:52:59.398562 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:52:59.398536    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e567580_1d49_4824_ac97_23eff3ab24b0.slice/crio-5fcc3dfa72d834652e01816774a96eab4bc603de9bcc3df3fb65d95f0ed8ec77 WatchSource:0}: Error finding container 5fcc3dfa72d834652e01816774a96eab4bc603de9bcc3df3fb65d95f0ed8ec77: Status 404 returned error can't find the container with id 5fcc3dfa72d834652e01816774a96eab4bc603de9bcc3df3fb65d95f0ed8ec77
Oct 30 14:52:59.400657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.400628    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:59.400713 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:52:59.400695    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:59.638028 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:52:59.637993    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:53:00.334230 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.334192    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" event={"ID":"0e567580-1d49-4824-ac97-23eff3ab24b0","Type":"ContainerStarted","Data":"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"}
Oct 30 14:53:00.334683 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.334247    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" event={"ID":"0e567580-1d49-4824-ac97-23eff3ab24b0","Type":"ContainerStarted","Data":"5fcc3dfa72d834652e01816774a96eab4bc603de9bcc3df3fb65d95f0ed8ec77"}
Oct 30 14:53:00.334683 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.334428    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:53:00.335889 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.335865    5064 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vbpm8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:53:00.336023 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.335912    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:53:00.388958 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:00.388900    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" podStartSLOduration=2.38888172 podStartE2EDuration="2.38888172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:00.386868233 +0000 UTC m=+329.828758959" watchObservedRunningTime="2026-10-30 14:53:00.38888172 +0000 UTC m=+329.830772437"
Oct 30 14:53:01.339950 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.339919    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 14:53:01.441015 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.440981    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:53:01.441251 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.441194    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery" containerID="cri-o://a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47" gracePeriod=30
Oct 30 14:53:01.683342 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.683311    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:53:01.736561 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736529    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736719 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736576    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736719 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736609    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736719 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736642    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736719 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736675    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736951 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736797    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.736951 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.736902    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\") pod \"877d2d9d-98f2-4731-be74-0cc6ef486136\" (UID: \"877d2d9d-98f2-4731-be74-0cc6ef486136\") "
Oct 30 14:53:01.737293 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.737118    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:53:01.737293 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.737265    5064 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-ca-configmap\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.742927 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.742893    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:53:01.742927 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.742902    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:53:01.743111 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.742994    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:53:01.743111 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.742998    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "kube-api-access-2hw2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:53:01.743275 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.743264    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:53:01.743651 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.743629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs" pod "877d2d9d-98f2-4731-be74-0cc6ef486136" (UID: "877d2d9d-98f2-4731-be74-0cc6ef486136"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:53:01.838290 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838238    5064 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-kubeconfig\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.838457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838373    5064 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-csr-dns-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.838457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838403    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hw2j\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-kube-api-access-2hw2j\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.838457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838419    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877d2d9d-98f2-4731-be74-0cc6ef486136-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.838457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838433    5064 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/877d2d9d-98f2-4731-be74-0cc6ef486136-cacerts\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:01.838457 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:01.838446    5064 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/877d2d9d-98f2-4731-be74-0cc6ef486136-local-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:53:02.346029 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.345993    5064 generic.go:356] "Generic (PLEG): container finished" podID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerID="a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47" exitCode=0
Oct 30 14:53:02.346510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.346056    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"
Oct 30 14:53:02.346510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.346078    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" event={"ID":"877d2d9d-98f2-4731-be74-0cc6ef486136","Type":"ContainerDied","Data":"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"}
Oct 30 14:53:02.346510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.346160    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7" event={"ID":"877d2d9d-98f2-4731-be74-0cc6ef486136","Type":"ContainerDied","Data":"f8b45bb216e6c11d1d2bbd1ce5e7cc74a08dc235cd5dda5ea29f7aa0015ebec9"}
Oct 30 14:53:02.346510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.346181    5064 scope.go:122] "RemoveContainer" containerID="a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"
Oct 30 14:53:02.355170 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.355150    5064 scope.go:122] "RemoveContainer" containerID="a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"
Oct 30 14:53:02.355441 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:53:02.355420    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47\": container with ID starting with a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47 not found: ID does not exist" containerID="a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"
Oct 30 14:53:02.355488 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.355451    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47"} err="failed to get container status \"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47\": rpc error: code = NotFound desc = could not find container \"a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47\": container with ID starting with a46df00b7027386f187fef9afe67382870f722765d26efdd46d784eb2bf16e47 not found: ID does not exist"
Oct 30 14:53:02.402153 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.402124    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:53:02.416007 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.415980    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8r5f7"]
Oct 30 14:53:02.720578 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:53:02.720543    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" path="/var/lib/kubelet/pods/877d2d9d-98f2-4731-be74-0cc6ef486136/volumes"
Oct 30 14:53:29.629130 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:53:29.629093    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:53:59.629926 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:53:59.629885    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:54:28.872925 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.872884    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"]
Oct 30 14:54:28.873384 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.873264    5064 cpu_manager.go:414] "containerMap: removing container" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery"
Oct 30 14:54:28.873384 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.873275    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery"
Oct 30 14:54:28.873384 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.873334    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="877d2d9d-98f2-4731-be74-0cc6ef486136" containerName="discovery"
Oct 30 14:54:28.876263 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.876245    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:28.890226 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.890198    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"]
Oct 30 14:54:28.899989 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.899957    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2z8\" (UniqueName: \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:28.900172 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.900004    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:28.902590 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.902559    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 14:54:28.902590 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.902576    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 14:54:28.902737 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:28.902602    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wvm9s\""
Oct 30 14:54:29.001171 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.001133    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2z8\" (UniqueName: \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:29.001171 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.001175    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:29.001584 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.001563    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:29.029512 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.029476    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2z8\" (UniqueName: \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:29.190074 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.189982    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:29.522954 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.522856    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"]
Oct 30 14:54:29.537878 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:54:29.537814    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b72bd0d_eb4f_48e1_aeaf_d1bd84f69ff5.slice/crio-f1c089ecfeb0b4bfadb819ec4a56bcd0b1f0fefb8b1981b0b3118c2e008a9dde WatchSource:0}: Error finding container f1c089ecfeb0b4bfadb819ec4a56bcd0b1f0fefb8b1981b0b3118c2e008a9dde: Status 404 returned error can't find the container with id f1c089ecfeb0b4bfadb819ec4a56bcd0b1f0fefb8b1981b0b3118c2e008a9dde
Oct 30 14:54:29.542321 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.542299    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:54:29.629465 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:29.629423    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:54:29.723246 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:29.723212    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz" event={"ID":"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5","Type":"ContainerStarted","Data":"f1c089ecfeb0b4bfadb819ec4a56bcd0b1f0fefb8b1981b0b3118c2e008a9dde"}
Oct 30 14:54:30.767093 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:30.767058    5064 scope.go:122] "RemoveContainer" containerID="3f416c9d19632b75f6c681efdc7548bd2369ec292c9675d7cebff9fef8405718"
Oct 30 14:54:31.430328 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.430291    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"]
Oct 30 14:54:31.437682 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.437651    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:31.464872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.464781    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"]
Oct 30 14:54:31.476715 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.476676    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v58dp\""
Oct 30 14:54:31.524955 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.524910    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx84l\" (UniqueName: \"kubernetes.io/projected/599621e7-347f-4568-bead-91a824451b43-kube-api-access-mx84l\") pod \"authorino-operator-68d6dd76dc-fnqxl\" (UID: \"599621e7-347f-4568-bead-91a824451b43\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:31.625764 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.625730    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx84l\" (UniqueName: \"kubernetes.io/projected/599621e7-347f-4568-bead-91a824451b43-kube-api-access-mx84l\") pod \"authorino-operator-68d6dd76dc-fnqxl\" (UID: \"599621e7-347f-4568-bead-91a824451b43\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:31.655676 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.655636    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx84l\" (UniqueName: \"kubernetes.io/projected/599621e7-347f-4568-bead-91a824451b43-kube-api-access-mx84l\") pod \"authorino-operator-68d6dd76dc-fnqxl\" (UID: \"599621e7-347f-4568-bead-91a824451b43\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:31.755982 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:31.755903    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:32.092592 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:32.092564    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"]
Oct 30 14:54:32.094984 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:54:32.094948    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599621e7_347f_4568_bead_91a824451b43.slice/crio-001604e7f60837c9546d53646d794a505dbbe4270acf3995d7719db7c6bf6f8a WatchSource:0}: Error finding container 001604e7f60837c9546d53646d794a505dbbe4270acf3995d7719db7c6bf6f8a: Status 404 returned error can't find the container with id 001604e7f60837c9546d53646d794a505dbbe4270acf3995d7719db7c6bf6f8a
Oct 30 14:54:32.741317 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:32.741277    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl" event={"ID":"599621e7-347f-4568-bead-91a824451b43","Type":"ContainerStarted","Data":"001604e7f60837c9546d53646d794a505dbbe4270acf3995d7719db7c6bf6f8a"}
Oct 30 14:54:34.752549 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.752510    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz" event={"ID":"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5","Type":"ContainerStarted","Data":"ecc89809e642af4de5df17755f6f1f776be86bb700a6393b47bc713331af5876"}
Oct 30 14:54:34.753073 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.752587    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:34.754205 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.754173    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl" event={"ID":"599621e7-347f-4568-bead-91a824451b43","Type":"ContainerStarted","Data":"f258cacf2e63c6525bb6c8e7713eb79f80f2fa8c9b9f5daf9db39be341a1e8e8"}
Oct 30 14:54:34.754327 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.754306    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:34.833798 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.833744    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz" podStartSLOduration=2.589735347 podStartE2EDuration="6.833731503s" totalImagesPullingTime="4.243996156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:34.818804865 +0000 UTC m=+424.260695582" watchObservedRunningTime="2026-10-30 14:54:34.833731503 +0000 UTC m=+424.275622219"
Oct 30 14:54:34.868251 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:34.868206    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl" podStartSLOduration=2.254216635 podStartE2EDuration="3.868190002s" totalImagesPullingTime="1.613973367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:34.867655807 +0000 UTC m=+424.309546524" watchObservedRunningTime="2026-10-30 14:54:34.868190002 +0000 UTC m=+424.310080720"
Oct 30 14:54:45.761767 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:45.761736    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-fnqxl"
Oct 30 14:54:45.762205 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:45.761823    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:48.698198 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.698161    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"]
Oct 30 14:54:48.701662 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.701644    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.721208 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.721176    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"]
Oct 30 14:54:48.784429 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.784392    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wqd\" (UniqueName: \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.784595 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.784448    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.885587 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.885547    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wqd\" (UniqueName: \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.885781 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.885700    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.886143 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.886119    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:48.934746 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:48.934711    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wqd\" (UniqueName: \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\") pod \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:49.013583 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.013493    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:49.356475 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.356393    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"]
Oct 30 14:54:49.372871 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:54:49.372787    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a71dcc_ab49_4269_920a_21efd2661b61.slice/crio-127b23808370e5de746d55aabacce2a2266e2d8b6591ba26a6859f418f3ae3c2 WatchSource:0}: Error finding container 127b23808370e5de746d55aabacce2a2266e2d8b6591ba26a6859f418f3ae3c2: Status 404 returned error can't find the container with id 127b23808370e5de746d55aabacce2a2266e2d8b6591ba26a6859f418f3ae3c2
Oct 30 14:54:49.653324 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.653286    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"]
Oct 30 14:54:49.653504 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.653479    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" containerName="manager" containerID="cri-o://ecc89809e642af4de5df17755f6f1f776be86bb700a6393b47bc713331af5876" gracePeriod=2
Oct 30 14:54:49.669048 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.669014    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.670046 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.669983    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"]
Oct 30 14:54:49.683024 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.682986    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:54:49.683399 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.683383    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" containerName="manager"
Oct 30 14:54:49.683471 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.683400    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" containerName="manager"
Oct 30 14:54:49.683511 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.683505    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" containerName="manager"
Oct 30 14:54:49.686743 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.686718    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.708302 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.708269    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"]
Oct 30 14:54:49.715791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.715760    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:54:49.717875 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.717846    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"]
Oct 30 14:54:49.745106 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.745071    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:54:49.745888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.745863    5064 cpu_manager.go:414] "containerMap: removing container" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" containerName="manager"
Oct 30 14:54:49.745888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.745885    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" containerName="manager"
Oct 30 14:54:49.746500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.746476    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" containerName="manager"
Oct 30 14:54:49.749875 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.749852    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:49.763247 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.763213    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.780100 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.780067    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.787271 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.787233    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:54:49.796185 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.796149    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6t5\" (UniqueName: \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.796389 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.796232    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.824898 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.824847    5064 generic.go:356] "Generic (PLEG): container finished" podID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" containerID="ecc89809e642af4de5df17755f6f1f776be86bb700a6393b47bc713331af5876" exitCode=0
Oct 30 14:54:49.841547 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.841506    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.857641 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.857597    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:49.897101 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.897059    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.897262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.897132    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5295p\" (UniqueName: \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:49.897262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.897234    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:49.897355 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.897315    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6t5\" (UniqueName: \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.897505 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.897483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.906047 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.905990    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.921721 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.921687    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:49.926979 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.926959    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6t5\" (UniqueName: \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\") pod \"kuadrant-operator-controller-manager-6b6fb56846-ldpkr\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:49.936622 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:49.936591    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:49.998125 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998085    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\") pod \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") "
Oct 30 14:54:49.998314 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998186    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\" (UniqueName: \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\") pod \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\" (UID: \"2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5\") "
Oct 30 14:54:49.998480 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998458    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:49.998572 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5295p\" (UniqueName: \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:49.998721 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998700    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume" pod "2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" (UID: "2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:54:49.998927 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:49.998902    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:50.000525 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.000498    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8" pod "2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" (UID: "2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5"). InnerVolumeSpecName "kube-api-access-ms2z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:54:50.026727 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.026691    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5295p\" (UniqueName: \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\") pod \"kuadrant-operator-controller-manager-858bb878d8-nqm6l\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:50.036512 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.036485    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:50.065649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.065619    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:50.099566 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.099539    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2z8\" (UniqueName: \"kubernetes.io/projected/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-kube-api-access-ms2z8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:54:50.099693 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.099580    5064 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5-extensions-socket-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:54:50.351991 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.351968    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:54:50.355033 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:54:50.355001    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066f54b3_98a7_40f7_81dd_c0190ff8736b.slice/crio-9bef125428c04114860d24f0b4ed93efc9ddb5d4fab6ed7159461f2c273d4e3c WatchSource:0}: Error finding container 9bef125428c04114860d24f0b4ed93efc9ddb5d4fab6ed7159461f2c273d4e3c: Status 404 returned error can't find the container with id 9bef125428c04114860d24f0b4ed93efc9ddb5d4fab6ed7159461f2c273d4e3c
Oct 30 14:54:50.374545 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.374519    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:54:50.383897 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:54:50.383867    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf99581_8e3f_4c58_bd81_46f56f9b1a9b.slice/crio-18304a4c568a6b52769eb0354233eba6129cfceb9dfbaa9620ce03b8a96f5228 WatchSource:0}: Error finding container 18304a4c568a6b52769eb0354233eba6129cfceb9dfbaa9620ce03b8a96f5228: Status 404 returned error can't find the container with id 18304a4c568a6b52769eb0354233eba6129cfceb9dfbaa9620ce03b8a96f5228
Oct 30 14:54:50.721042 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.721007    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" path="/var/lib/kubelet/pods/2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5/volumes"
Oct 30 14:54:50.732007 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:50.731965    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-lqnhz\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="2b72bd0d-eb4f-48e1-aeaf-d1bd84f69ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:50.746869 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:50.746819    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:50.831199 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.831153    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" event={"ID":"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b","Type":"ContainerStarted","Data":"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"}
Oct 30 14:54:50.831409 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.831236    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:54:50.831409 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.831264    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" event={"ID":"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b","Type":"ContainerStarted","Data":"18304a4c568a6b52769eb0354233eba6129cfceb9dfbaa9620ce03b8a96f5228"}
Oct 30 14:54:50.832890 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.832859    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" event={"ID":"066f54b3-98a7-40f7-81dd-c0190ff8736b","Type":"ContainerStarted","Data":"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"}
Oct 30 14:54:50.833032 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.832915    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" event={"ID":"066f54b3-98a7-40f7-81dd-c0190ff8736b","Type":"ContainerStarted","Data":"9bef125428c04114860d24f0b4ed93efc9ddb5d4fab6ed7159461f2c273d4e3c"}
Oct 30 14:54:50.833032 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.832978    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:54:50.834321 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.834304    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lqnhz"
Oct 30 14:54:50.834412 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.834347    5064 scope.go:122] "RemoveContainer" containerID="ecc89809e642af4de5df17755f6f1f776be86bb700a6393b47bc713331af5876"
Oct 30 14:54:50.834499 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:50.834451    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" containerName="manager" containerID="cri-o://44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931" gracePeriod=2
Oct 30 14:54:51.001503 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.001376    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" podStartSLOduration=2.001361664 podStartE2EDuration="2.001361664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:50.942244157 +0000 UTC m=+440.384134875" watchObservedRunningTime="2026-10-30 14:54:51.001361664 +0000 UTC m=+440.443252382"
Oct 30 14:54:51.002097 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.002058    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" podStartSLOduration=2.002047928 podStartE2EDuration="2.002047928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:51.000188436 +0000 UTC m=+440.442079174" watchObservedRunningTime="2026-10-30 14:54:51.002047928 +0000 UTC m=+440.443938644"
Oct 30 14:54:51.099733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.099706    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:51.115096 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:51.115046    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:51.210382 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.210343    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\" (UniqueName: \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\") pod \"28a71dcc-ab49-4269-920a-21efd2661b61\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") "
Oct 30 14:54:51.210584 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.210409    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\") pod \"28a71dcc-ab49-4269-920a-21efd2661b61\" (UID: \"28a71dcc-ab49-4269-920a-21efd2661b61\") "
Oct 30 14:54:51.210718 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.210686    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume" pod "28a71dcc-ab49-4269-920a-21efd2661b61" (UID: "28a71dcc-ab49-4269-920a-21efd2661b61"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:54:51.212547 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.212517    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd" pod "28a71dcc-ab49-4269-920a-21efd2661b61" (UID: "28a71dcc-ab49-4269-920a-21efd2661b61"). InnerVolumeSpecName "kube-api-access-76wqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:54:51.311649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.311549    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wqd\" (UniqueName: \"kubernetes.io/projected/28a71dcc-ab49-4269-920a-21efd2661b61-kube-api-access-76wqd\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:54:51.311649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.311583    5064 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a71dcc-ab49-4269-920a-21efd2661b61-extensions-socket-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:54:51.841662 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.841625    5064 generic.go:356] "Generic (PLEG): container finished" podID="28a71dcc-ab49-4269-920a-21efd2661b61" containerID="44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931" exitCode=2
Oct 30 14:54:51.842151 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.841672    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:51.842151 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.841722    5064 scope.go:122] "RemoveContainer" containerID="44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931"
Oct 30 14:54:51.850587 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.850567    5064 scope.go:122] "RemoveContainer" containerID="44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931"
Oct 30 14:54:51.850879 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:51.850858    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931\": container with ID starting with 44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931 not found: ID does not exist" containerID="44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931"
Oct 30 14:54:51.850936 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:51.850890    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931"} err="failed to get container status \"44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931\": rpc error: code = NotFound desc = could not find container \"44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931\": container with ID starting with 44d52baf316641f29a797e66e17a0fa6912c571409a37970fe23fc83bad07931 not found: ID does not exist"
Oct 30 14:54:51.857019 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:51.856990    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:51.871978 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:51.871942    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-q97k4\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-q97k4"
Oct 30 14:54:52.721590 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:54:52.721551    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a71dcc-ab49-4269-920a-21efd2661b61" path="/var/lib/kubelet/pods/28a71dcc-ab49-4269-920a-21efd2661b61/volumes"
Oct 30 14:54:59.629552 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:54:59.629513    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:55:01.844116 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:01.844084    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:55:01.844537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:01.844263    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:55:01.985712 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:01.985675    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:55:01.985920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:01.985882    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" podUID="066f54b3-98a7-40f7-81dd-c0190ff8736b" containerName="manager" containerID="cri-o://a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6" gracePeriod=10
Oct 30 14:55:02.269610 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.269582    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:55:02.410799 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.410716    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\" (UniqueName: \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\") pod \"066f54b3-98a7-40f7-81dd-c0190ff8736b\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") "
Oct 30 14:55:02.410799 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.410760    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\") pod \"066f54b3-98a7-40f7-81dd-c0190ff8736b\" (UID: \"066f54b3-98a7-40f7-81dd-c0190ff8736b\") "
Oct 30 14:55:02.411207 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.411177    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume" pod "066f54b3-98a7-40f7-81dd-c0190ff8736b" (UID: "066f54b3-98a7-40f7-81dd-c0190ff8736b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:55:02.412968 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.412929    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5" pod "066f54b3-98a7-40f7-81dd-c0190ff8736b" (UID: "066f54b3-98a7-40f7-81dd-c0190ff8736b"). InnerVolumeSpecName "kube-api-access-dz6t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:55:02.511623 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.511591    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6t5\" (UniqueName: \"kubernetes.io/projected/066f54b3-98a7-40f7-81dd-c0190ff8736b-kube-api-access-dz6t5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:02.511623 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.511619    5064 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/066f54b3-98a7-40f7-81dd-c0190ff8736b-extensions-socket-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:02.895307 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.895274    5064 generic.go:356] "Generic (PLEG): container finished" podID="066f54b3-98a7-40f7-81dd-c0190ff8736b" containerID="a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6" exitCode=0
Oct 30 14:55:02.895769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.895395    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"
Oct 30 14:55:02.895769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.895407    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" event={"ID":"066f54b3-98a7-40f7-81dd-c0190ff8736b","Type":"ContainerDied","Data":"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"}
Oct 30 14:55:02.895769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.895469    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr" event={"ID":"066f54b3-98a7-40f7-81dd-c0190ff8736b","Type":"ContainerDied","Data":"9bef125428c04114860d24f0b4ed93efc9ddb5d4fab6ed7159461f2c273d4e3c"}
Oct 30 14:55:02.895769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.895500    5064 scope.go:122] "RemoveContainer" containerID="a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"
Oct 30 14:55:02.903859 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.903813    5064 scope.go:122] "RemoveContainer" containerID="a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"
Oct 30 14:55:02.904138 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:55:02.904120    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6\": container with ID starting with a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6 not found: ID does not exist" containerID="a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"
Oct 30 14:55:02.904188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.904147    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6"} err="failed to get container status \"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6\": rpc error: code = NotFound desc = could not find container \"a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6\": container with ID starting with a29361df39a4115556c8faaf56fd4a371fd8f0fff4a52fbdb2eee6a93b078ee6 not found: ID does not exist"
Oct 30 14:55:02.960657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.960624    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:55:02.974334 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:02.974305    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-ldpkr"]
Oct 30 14:55:04.720569 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:04.720529    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066f54b3-98a7-40f7-81dd-c0190ff8736b" path="/var/lib/kubelet/pods/066f54b3-98a7-40f7-81dd-c0190ff8736b/volumes"
Oct 30 14:55:15.059163 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.059122    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:55:15.059609 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.059362    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" podUID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" containerName="manager" containerID="cri-o://41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1" gracePeriod=10
Oct 30 14:55:15.312147 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.312082    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:55:15.427551 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.427506    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\") pod \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") "
Oct 30 14:55:15.427734 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.427559    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\" (UniqueName: \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\") pod \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\" (UID: \"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b\") "
Oct 30 14:55:15.427997 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.427971    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume" pod "dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" (UID: "dbf99581-8e3f-4c58-bd81-46f56f9b1a9b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:55:15.429699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.429671    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p" pod "dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" (UID: "dbf99581-8e3f-4c58-bd81-46f56f9b1a9b"). InnerVolumeSpecName "kube-api-access-5295p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:55:15.528456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.528417    5064 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-extensions-socket-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:15.528456 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.528447    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5295p\" (UniqueName: \"kubernetes.io/projected/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b-kube-api-access-5295p\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:15.956397 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.956360    5064 generic.go:356] "Generic (PLEG): container finished" podID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" containerID="41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1" exitCode=0
Oct 30 14:55:15.956600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.956414    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"
Oct 30 14:55:15.956600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.956442    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" event={"ID":"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b","Type":"ContainerDied","Data":"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"}
Oct 30 14:55:15.956600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.956504    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l" event={"ID":"dbf99581-8e3f-4c58-bd81-46f56f9b1a9b","Type":"ContainerDied","Data":"18304a4c568a6b52769eb0354233eba6129cfceb9dfbaa9620ce03b8a96f5228"}
Oct 30 14:55:15.956600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.956528    5064 scope.go:122] "RemoveContainer" containerID="41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"
Oct 30 14:55:15.965642 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.965626    5064 scope.go:122] "RemoveContainer" containerID="41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"
Oct 30 14:55:15.965919 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:55:15.965899    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1\": container with ID starting with 41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1 not found: ID does not exist" containerID="41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"
Oct 30 14:55:15.965965 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:15.965929    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1"} err="failed to get container status \"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1\": rpc error: code = NotFound desc = could not find container \"41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1\": container with ID starting with 41f403a16354c956ed1205b27995b3b53c9a732b5a637c5b2801017dc9345ea1 not found: ID does not exist"
Oct 30 14:55:16.034344 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:16.034306    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:55:16.039415 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:16.039386    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-nqm6l"]
Oct 30 14:55:16.721044 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:16.721010    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" path="/var/lib/kubelet/pods/dbf99581-8e3f-4c58-bd81-46f56f9b1a9b/volumes"
Oct 30 14:55:19.286698 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.286663    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"]
Oct 30 14:55:19.287112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287076    5064 cpu_manager.go:414] "containerMap: removing container" podUID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" containerName="manager"
Oct 30 14:55:19.287112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287089    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" containerName="manager"
Oct 30 14:55:19.287112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287104    5064 cpu_manager.go:414] "containerMap: removing container" podUID="066f54b3-98a7-40f7-81dd-c0190ff8736b" containerName="manager"
Oct 30 14:55:19.287112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287109    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066f54b3-98a7-40f7-81dd-c0190ff8736b" containerName="manager"
Oct 30 14:55:19.287245 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287161    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf99581-8e3f-4c58-bd81-46f56f9b1a9b" containerName="manager"
Oct 30 14:55:19.287245 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.287172    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="066f54b3-98a7-40f7-81dd-c0190ff8736b" containerName="manager"
Oct 30 14:55:19.290120 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.290100    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.315290 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.315260    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"]
Oct 30 14:55:19.361450 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361403    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361450 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361448    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361729 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361729 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361666    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5mw\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-kube-api-access-qm5mw\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361823 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361739    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361823 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361796    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361978 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361859    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.361978 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.361917    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.362090 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.362006    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.365013 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.364995    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khr2k\""
Oct 30 14:55:19.462551 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462521    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.462763 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462576    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.462763 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462608    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.462763 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462631    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.462763 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462728    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.462763 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462756    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5mw\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-kube-api-access-qm5mw\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462781    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462812    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.462881    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.463041    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463391 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.463346    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463499 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.463389    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463563 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.463530    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.463622 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.463583    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.465032 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.465008    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.465550 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.465531    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.488114 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.488070    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.488281 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.488179    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5mw\" (UniqueName: \"kubernetes.io/projected/982473c9-6215-4df5-8a3f-6d88ffbac6e4-kube-api-access-qm5mw\") pod \"maas-default-gateway-openshift-default-6cc458d887-9vjxl\" (UID: \"982473c9-6215-4df5-8a3f-6d88ffbac6e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.603500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.603397    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:19.923281 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:55:19.923231    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982473c9_6215_4df5_8a3f_6d88ffbac6e4.slice/crio-9537698cfc2d86dff655e34acd7cf2766dbf8916e9ec9939e9676295e05e82f5 WatchSource:0}: Error finding container 9537698cfc2d86dff655e34acd7cf2766dbf8916e9ec9939e9676295e05e82f5: Status 404 returned error can't find the container with id 9537698cfc2d86dff655e34acd7cf2766dbf8916e9ec9939e9676295e05e82f5
Oct 30 14:55:19.923501 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.923479    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"]
Oct 30 14:55:19.925701 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.925664    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:55:19.925809 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.925752    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:55:19.925809 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.925799    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:55:19.978640 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:19.978573    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" event={"ID":"982473c9-6215-4df5-8a3f-6d88ffbac6e4","Type":"ContainerStarted","Data":"9537698cfc2d86dff655e34acd7cf2766dbf8916e9ec9939e9676295e05e82f5"}
Oct 30 14:55:20.984950 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:20.984909    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" event={"ID":"982473c9-6215-4df5-8a3f-6d88ffbac6e4","Type":"ContainerStarted","Data":"2d5ddfdf869a81b8cd705de750324e124355cb70f3fdd8ce788dde0b1c4da529"}
Oct 30 14:55:21.075136 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:21.075079    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podStartSLOduration=2.075065481 podStartE2EDuration="2.075065481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:21.058475932 +0000 UTC m=+470.500366648" watchObservedRunningTime="2026-10-30 14:55:21.075065481 +0000 UTC m=+470.516956198"
Oct 30 14:55:21.603606 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:21.603567    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:21.609471 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:21.609439    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:21.991183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:21.991146    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:21.992007 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:21.991986    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:55:29.629619 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:55:29.629577    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:55:32.121664 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.121623    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:32.126574 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.126548    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.144377 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.144312    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 14:55:32.146252 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.146221    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:32.158010 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.157971    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl9hw\""
Oct 30 14:55:32.180137 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.180104    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtffr\" (UniqueName: \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.180306 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.180144    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.266027 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.265993    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:32.281350 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.281303    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtffr\" (UniqueName: \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.281350 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.281348    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.282045 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.282023    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.306270 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.306237    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtffr\" (UniqueName: \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\") pod \"limitador-limitador-74dfcfb894-cswv7\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.439498 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.439405    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:32.740710 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:55:32.740679    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ecb9db_a5e9_40c2_912c_47a88714ac65.slice/crio-f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef WatchSource:0}: Error finding container f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef: Status 404 returned error can't find the container with id f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef
Oct 30 14:55:32.750377 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:32.750351    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:33.043819 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:33.043734    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" event={"ID":"45ecb9db-a5e9-40c2-912c-47a88714ac65","Type":"ContainerStarted","Data":"f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef"}
Oct 30 14:55:35.054571 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:35.054531    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" event={"ID":"45ecb9db-a5e9-40c2-912c-47a88714ac65","Type":"ContainerStarted","Data":"4d37f820f204bc1bdd7e2eb29f1346be3b952d5132596d96f0eae0e128224a5d"}
Oct 30 14:55:35.055043 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:35.054676    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:35.099449 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:35.099387    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" podStartSLOduration=1.643505524 podStartE2EDuration="3.09937496s" totalImagesPullingTime="1.455869436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:35.098476819 +0000 UTC m=+484.540367539" watchObservedRunningTime="2026-10-30 14:55:35.09937496 +0000 UTC m=+484.541265713"
Oct 30 14:55:46.059262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:46.059185    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:47.184688 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:47.184654    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:47.185154 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:47.184878    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" podUID="45ecb9db-a5e9-40c2-912c-47a88714ac65" containerName="limitador" containerID="cri-o://4d37f820f204bc1bdd7e2eb29f1346be3b952d5132596d96f0eae0e128224a5d" gracePeriod=30
Oct 30 14:55:48.119582 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.119551    5064 generic.go:356] "Generic (PLEG): container finished" podID="45ecb9db-a5e9-40c2-912c-47a88714ac65" containerID="4d37f820f204bc1bdd7e2eb29f1346be3b952d5132596d96f0eae0e128224a5d" exitCode=0
Oct 30 14:55:48.119707 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.119613    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" event={"ID":"45ecb9db-a5e9-40c2-912c-47a88714ac65","Type":"ContainerDied","Data":"4d37f820f204bc1bdd7e2eb29f1346be3b952d5132596d96f0eae0e128224a5d"}
Oct 30 14:55:48.119707 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.119671    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7" event={"ID":"45ecb9db-a5e9-40c2-912c-47a88714ac65","Type":"ContainerDied","Data":"f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef"}
Oct 30 14:55:48.119707 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.119688    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f966b2e86ffb0d46c2bc106521847b5880592e1e36cdd2d0b63da225e2d400ef"
Oct 30 14:55:48.131233 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.131213    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:48.225139 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.225100    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\" (UniqueName: \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\") pod \"45ecb9db-a5e9-40c2-912c-47a88714ac65\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") "
Oct 30 14:55:48.225555 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.225167    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\" (UniqueName: \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\") pod \"45ecb9db-a5e9-40c2-912c-47a88714ac65\" (UID: \"45ecb9db-a5e9-40c2-912c-47a88714ac65\") "
Oct 30 14:55:48.225604 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.225538    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file" pod "45ecb9db-a5e9-40c2-912c-47a88714ac65" (UID: "45ecb9db-a5e9-40c2-912c-47a88714ac65"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:55:48.227298 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.227271    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr" pod "45ecb9db-a5e9-40c2-912c-47a88714ac65" (UID: "45ecb9db-a5e9-40c2-912c-47a88714ac65"). InnerVolumeSpecName "kube-api-access-qtffr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:55:48.326029 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.325936    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtffr\" (UniqueName: \"kubernetes.io/projected/45ecb9db-a5e9-40c2-912c-47a88714ac65-kube-api-access-qtffr\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:48.326029 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:48.325968    5064 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45ecb9db-a5e9-40c2-912c-47a88714ac65-config-file\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:55:49.124508 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:49.124474    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cswv7"
Oct 30 14:55:49.179653 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:49.179618    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:49.193317 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:49.193289    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cswv7"]
Oct 30 14:55:50.227917 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.227881    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"]
Oct 30 14:55:50.228484 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.228463    5064 cpu_manager.go:414] "containerMap: removing container" podUID="45ecb9db-a5e9-40c2-912c-47a88714ac65" containerName="limitador"
Oct 30 14:55:50.228573 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.228487    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ecb9db-a5e9-40c2-912c-47a88714ac65" containerName="limitador"
Oct 30 14:55:50.228627 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.228581    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ecb9db-a5e9-40c2-912c-47a88714ac65" containerName="limitador"
Oct 30 14:55:50.231886 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.231862    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.248528 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.248498    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"]
Oct 30 14:55:50.250279 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.250254    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:55:50.260098 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.260071    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:55:50.260098 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.260082    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 14:55:50.262818 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.262795    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fs7bb\""
Oct 30 14:55:50.345588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.345543    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/91d59170-b98e-471e-8dbb-ac0dfc15c071-data\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.345588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.345590    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752kd\" (UniqueName: \"kubernetes.io/projected/91d59170-b98e-471e-8dbb-ac0dfc15c071-kube-api-access-752kd\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.446937 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.446889    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/91d59170-b98e-471e-8dbb-ac0dfc15c071-data\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.446937 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.446937    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752kd\" (UniqueName: \"kubernetes.io/projected/91d59170-b98e-471e-8dbb-ac0dfc15c071-kube-api-access-752kd\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.447342 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.447321    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/91d59170-b98e-471e-8dbb-ac0dfc15c071-data\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.472475 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.472437    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752kd\" (UniqueName: \"kubernetes.io/projected/91d59170-b98e-471e-8dbb-ac0dfc15c071-kube-api-access-752kd\") pod \"postgres-7cdbf6dc44-46kbm\" (UID: \"91d59170-b98e-471e-8dbb-ac0dfc15c071\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.544801 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.544698    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:50.720630 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.720584    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ecb9db-a5e9-40c2-912c-47a88714ac65" path="/var/lib/kubelet/pods/45ecb9db-a5e9-40c2-912c-47a88714ac65/volumes"
Oct 30 14:55:50.841625 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:50.841481    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"]
Oct 30 14:55:50.844936 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:55:50.844902    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d59170_b98e_471e_8dbb_ac0dfc15c071.slice/crio-40d964b02e646c6bdbf3bd6611992c69a35be660c245aa5638bcf601ad07b87a WatchSource:0}: Error finding container 40d964b02e646c6bdbf3bd6611992c69a35be660c245aa5638bcf601ad07b87a: Status 404 returned error can't find the container with id 40d964b02e646c6bdbf3bd6611992c69a35be660c245aa5638bcf601ad07b87a
Oct 30 14:55:51.134352 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:51.134319    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm" event={"ID":"91d59170-b98e-471e-8dbb-ac0dfc15c071","Type":"ContainerStarted","Data":"40d964b02e646c6bdbf3bd6611992c69a35be660c245aa5638bcf601ad07b87a"}
Oct 30 14:55:56.160399 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:56.160358    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm" event={"ID":"91d59170-b98e-471e-8dbb-ac0dfc15c071","Type":"ContainerStarted","Data":"d26bfd6015fa3bc16f74ecbf22d9c17d6d2fe3e7fb04ee25f101ad4cfc8f0ac7"}
Oct 30 14:55:56.160802 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:56.160542    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:55:56.204295 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:55:56.204241    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm" podStartSLOduration=1.277378785 podStartE2EDuration="6.204224974s" totalImagesPullingTime="4.926846189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:56.202773251 +0000 UTC m=+505.644663970" watchObservedRunningTime="2026-10-30 14:55:56.204224974 +0000 UTC m=+505.646115690"
Oct 30 14:55:59.628307 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:55:59.628268    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:56:02.194612 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:02.194580    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-46kbm"
Oct 30 14:56:12.659608 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.659571    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:12.663238 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.663215    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.705182 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.705151    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rwtwc\""
Oct 30 14:56:12.705182 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.705151    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 14:56:12.705182 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.705151    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 14:56:12.705473 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.705203    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 14:56:12.706980 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.706953    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:12.740481 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.740444    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.740657 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.740564    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tr6\" (UniqueName: \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.740739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.740654    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.841937 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.841897    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tr6\" (UniqueName: \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.842107 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.842073    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.842158 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.842135    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.844561 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.844536    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.844665 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.844565    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.866988 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.866960    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tr6\" (UniqueName: \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\") pod \"maas-controller-577f596567-kdzvc\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") " pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:12.975371 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:12.975282    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:13.067743 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.067713    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 14:56:13.071981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.071960    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.096543 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.096510    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 14:56:13.145137 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.145095    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.145137 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.145135    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.145359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.145168    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjqm\" (UniqueName: \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.246238 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.246145    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.246238 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.246186    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.246238 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.246231    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjqm\" (UniqueName: \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.248701 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.248670    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.248865 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.248728    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.271769 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.271733    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjqm\" (UniqueName: \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\") pod \"maas-controller-68588d554c-9fpwf\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") " pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.275102 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.275080    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:13.277691 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:56:13.277663    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a41a5d_d960_4029_9eb0_51fc0e23862f.slice/crio-90b3f4276e25884c4aad16450ebf19c13951eb78d00ef984323af5ff89427750 WatchSource:0}: Error finding container 90b3f4276e25884c4aad16450ebf19c13951eb78d00ef984323af5ff89427750: Status 404 returned error can't find the container with id 90b3f4276e25884c4aad16450ebf19c13951eb78d00ef984323af5ff89427750
Oct 30 14:56:13.385185 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.385148    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:13.684175 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:13.684132    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 14:56:13.686793 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:56:13.686756    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d3e086_bc80_43e2_aba5_190c8ff3cc4a.slice/crio-5127e828f674ba1f3a02e7a98581eb2ef779cf26df8dba888a4a2e2a4e2254cf WatchSource:0}: Error finding container 5127e828f674ba1f3a02e7a98581eb2ef779cf26df8dba888a4a2e2a4e2254cf: Status 404 returned error can't find the container with id 5127e828f674ba1f3a02e7a98581eb2ef779cf26df8dba888a4a2e2a4e2254cf
Oct 30 14:56:14.248871 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:14.248777    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-kdzvc" event={"ID":"07a41a5d-d960-4029-9eb0-51fc0e23862f","Type":"ContainerStarted","Data":"90b3f4276e25884c4aad16450ebf19c13951eb78d00ef984323af5ff89427750"}
Oct 30 14:56:14.250303 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:14.250212    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9fpwf" event={"ID":"46d3e086-bc80-43e2-aba5-190c8ff3cc4a","Type":"ContainerStarted","Data":"5127e828f674ba1f3a02e7a98581eb2ef779cf26df8dba888a4a2e2a4e2254cf"}
Oct 30 14:56:16.262194 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:16.262150    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-kdzvc" event={"ID":"07a41a5d-d960-4029-9eb0-51fc0e23862f","Type":"ContainerStarted","Data":"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"}
Oct 30 14:56:16.262592 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:16.262347    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:16.307946 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:16.307898    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-kdzvc" podStartSLOduration=1.444964524 podStartE2EDuration="4.30788349s" totalImagesPullingTime="2.862918966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:16.307325652 +0000 UTC m=+525.749216394" watchObservedRunningTime="2026-10-30 14:56:16.30788349 +0000 UTC m=+525.749774198"
Oct 30 14:56:17.268190 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:17.268145    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9fpwf" event={"ID":"46d3e086-bc80-43e2-aba5-190c8ff3cc4a","Type":"ContainerStarted","Data":"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"}
Oct 30 14:56:17.268610 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:17.268308    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:17.320273 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:17.320215    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68588d554c-9fpwf" podStartSLOduration=1.7152452440000001 podStartE2EDuration="4.320200982s" totalImagesPullingTime="2.604955738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:17.317884991 +0000 UTC m=+526.759775705" watchObservedRunningTime="2026-10-30 14:56:17.320200982 +0000 UTC m=+526.762091698"
Oct 30 14:56:20.051541 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.051505    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"]
Oct 30 14:56:20.055386 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.055364    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.073812 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.073781    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 14:56:20.074124 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.073893    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rw5dq\""
Oct 30 14:56:20.074124 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.073893    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 14:56:20.089183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.087440    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 14:56:20.089183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.089111    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"]
Oct 30 14:56:20.114889 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.114849    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.115050 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.114967    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7505b204-f9df-471b-94b8-546ae5d5c907-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.115050 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.115010    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphvr\" (UniqueName: \"kubernetes.io/projected/7505b204-f9df-471b-94b8-546ae5d5c907-kube-api-access-hphvr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.115050 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.115042    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.215673 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.215640    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7505b204-f9df-471b-94b8-546ae5d5c907-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.215890 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.215680    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphvr\" (UniqueName: \"kubernetes.io/projected/7505b204-f9df-471b-94b8-546ae5d5c907-kube-api-access-hphvr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.215890 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.215713    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.215890 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.215815    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.216481 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.216441    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.216641 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.216620    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7505b204-f9df-471b-94b8-546ae5d5c907-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.218234 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.218215    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7505b204-f9df-471b-94b8-546ae5d5c907-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.240718 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.240692    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphvr\" (UniqueName: \"kubernetes.io/projected/7505b204-f9df-471b-94b8-546ae5d5c907-kube-api-access-hphvr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-5dxn5\" (UID: \"7505b204-f9df-471b-94b8-546ae5d5c907\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.369006 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.368979    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:20.668179 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:20.668150    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"]
Oct 30 14:56:20.670736 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:56:20.670705    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7505b204_f9df_471b_94b8_546ae5d5c907.slice/crio-6138fc7584fef033396c9c4f8b2f0e222256665aaa1773f7659ddd4a1270d397 WatchSource:0}: Error finding container 6138fc7584fef033396c9c4f8b2f0e222256665aaa1773f7659ddd4a1270d397: Status 404 returned error can't find the container with id 6138fc7584fef033396c9c4f8b2f0e222256665aaa1773f7659ddd4a1270d397
Oct 30 14:56:21.288176 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:21.288133    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5" event={"ID":"7505b204-f9df-471b-94b8-546ae5d5c907","Type":"ContainerStarted","Data":"6138fc7584fef033396c9c4f8b2f0e222256665aaa1773f7659ddd4a1270d397"}
Oct 30 14:56:27.271052 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:27.271014    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:28.276287 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.276249    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 14:56:28.354428 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.354390    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:28.354700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.354634    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-kdzvc" podUID="07a41a5d-d960-4029-9eb0-51fc0e23862f" containerName="manager" containerID="cri-o://08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14" gracePeriod=30
Oct 30 14:56:28.632748 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.632720    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:28.697798 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.697766    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\") pod \"07a41a5d-d960-4029-9eb0-51fc0e23862f\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") "
Oct 30 14:56:28.697995 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.697879    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\") pod \"07a41a5d-d960-4029-9eb0-51fc0e23862f\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") "
Oct 30 14:56:28.697995 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.697939    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\" (UniqueName: \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\") pod \"07a41a5d-d960-4029-9eb0-51fc0e23862f\" (UID: \"07a41a5d-d960-4029-9eb0-51fc0e23862f\") "
Oct 30 14:56:28.700958 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.700924    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert" pod "07a41a5d-d960-4029-9eb0-51fc0e23862f" (UID: "07a41a5d-d960-4029-9eb0-51fc0e23862f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:56:28.701104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.700960    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6" pod "07a41a5d-d960-4029-9eb0-51fc0e23862f" (UID: "07a41a5d-d960-4029-9eb0-51fc0e23862f"). InnerVolumeSpecName "kube-api-access-g8tr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:56:28.701104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.701054    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs" pod "07a41a5d-d960-4029-9eb0-51fc0e23862f" (UID: "07a41a5d-d960-4029-9eb0-51fc0e23862f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:56:28.799713 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.799625    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:56:28.799713 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.799666    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tr6\" (UniqueName: \"kubernetes.io/projected/07a41a5d-d960-4029-9eb0-51fc0e23862f-kube-api-access-g8tr6\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:56:28.799713 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:28.799679    5064 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a41a5d-d960-4029-9eb0-51fc0e23862f-webhook-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:56:29.339211 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.339167    5064 generic.go:356] "Generic (PLEG): container finished" podID="07a41a5d-d960-4029-9eb0-51fc0e23862f" containerID="08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14" exitCode=0
Oct 30 14:56:29.339699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.339243    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-kdzvc"
Oct 30 14:56:29.339699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.339257    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-kdzvc" event={"ID":"07a41a5d-d960-4029-9eb0-51fc0e23862f","Type":"ContainerDied","Data":"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"}
Oct 30 14:56:29.339699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.339328    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-kdzvc" event={"ID":"07a41a5d-d960-4029-9eb0-51fc0e23862f","Type":"ContainerDied","Data":"90b3f4276e25884c4aad16450ebf19c13951eb78d00ef984323af5ff89427750"}
Oct 30 14:56:29.339699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.339360    5064 scope.go:122] "RemoveContainer" containerID="08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"
Oct 30 14:56:29.351152 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.351126    5064 scope.go:122] "RemoveContainer" containerID="08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"
Oct 30 14:56:29.351514 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:56:29.351480    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14\": container with ID starting with 08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14 not found: ID does not exist" containerID="08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"
Oct 30 14:56:29.351604 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.351527    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14"} err="failed to get container status \"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14\": rpc error: code = NotFound desc = could not find container \"08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14\": container with ID starting with 08e11c5f952c9309ceb49a45086bf230be3ba502cd898cf497aa91daf3fb6f14 not found: ID does not exist"
Oct 30 14:56:29.395892 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.395853    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:29.409111 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:29.409075    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-kdzvc"]
Oct 30 14:56:29.635468 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:56:29.635424    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:56:30.722251 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:30.722217    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a41a5d-d960-4029-9eb0-51fc0e23862f" path="/var/lib/kubelet/pods/07a41a5d-d960-4029-9eb0-51fc0e23862f/volumes"
Oct 30 14:56:31.351933 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:31.351882    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5" event={"ID":"7505b204-f9df-471b-94b8-546ae5d5c907","Type":"ContainerStarted","Data":"74a86515ce50f6195d30b7b72687f9f629f3b39acb8c776890c70c92db40071e"}
Oct 30 14:56:31.352142 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:31.351954    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:31.398714 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:31.398657    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5" podStartSLOduration=0.899272371 podStartE2EDuration="11.398638781s" totalImagesPullingTime="10.49936641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:31.39694289 +0000 UTC m=+540.838833606" watchObservedRunningTime="2026-10-30 14:56:31.398638781 +0000 UTC m=+540.840529501"
Oct 30 14:56:42.361732 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:42.361698    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-5dxn5"
Oct 30 14:56:51.310060 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.310014    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 14:56:51.435825 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.310574    5064 cpu_manager.go:414] "containerMap: removing container" podUID="07a41a5d-d960-4029-9eb0-51fc0e23862f" containerName="manager"
Oct 30 14:56:51.435825 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.310592    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a41a5d-d960-4029-9eb0-51fc0e23862f" containerName="manager"
Oct 30 14:56:51.435825 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.310686    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a41a5d-d960-4029-9eb0-51fc0e23862f" containerName="manager"
Oct 30 14:56:51.442030 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.441986    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 14:56:51.442203 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.442146    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.476239 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.476203    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4jtfs\""
Oct 30 14:56:51.476239 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.476237    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 14:56:51.476443 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.476259    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 14:56:51.508890 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.508848    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtm7b\" (UniqueName: \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.509071 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.509005    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.509071 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.509044    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.509146 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.509072    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.610053 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.609978    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.610053 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.610023    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.610287 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.610056    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.610287 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.610121    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtm7b\" (UniqueName: \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.610868 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.610822    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.612573 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.612549    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.612698 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.612633    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.635932 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.635899    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtm7b\" (UniqueName: \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\") pod \"maas-api-5c48877c75-hlm6r\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:51.787349 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:51.787312    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:52.149685 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:52.149659    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 14:56:52.151477 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:56:52.151447    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1336714_66b2_4de3_9dc9_03828e5da7ac.slice/crio-6f6a22963a2c38bacd45c1c49c9731f11e1cda6365544b2e4122bbb4c8864f75 WatchSource:0}: Error finding container 6f6a22963a2c38bacd45c1c49c9731f11e1cda6365544b2e4122bbb4c8864f75: Status 404 returned error can't find the container with id 6f6a22963a2c38bacd45c1c49c9731f11e1cda6365544b2e4122bbb4c8864f75
Oct 30 14:56:52.449921 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:52.449810    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" event={"ID":"b1336714-66b2-4de3-9dc9-03828e5da7ac","Type":"ContainerStarted","Data":"6f6a22963a2c38bacd45c1c49c9731f11e1cda6365544b2e4122bbb4c8864f75"}
Oct 30 14:56:54.463866 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:54.463754    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" event={"ID":"b1336714-66b2-4de3-9dc9-03828e5da7ac","Type":"ContainerStarted","Data":"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"}
Oct 30 14:56:54.464320 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:54.463899    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:56:54.511054 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:56:54.510991    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" podStartSLOduration=1.516725629 podStartE2EDuration="3.510977007s" totalImagesPullingTime="1.994251378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:54.50980587 +0000 UTC m=+563.951696586" watchObservedRunningTime="2026-10-30 14:56:54.510977007 +0000 UTC m=+563.952867724"
Oct 30 14:56:59.632083 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:56:59.632027    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:57:00.474941 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:00.474902    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 14:57:03.508951 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.508914    5064 generic.go:356] "Generic (PLEG): container finished" podID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerID="2d5ddfdf869a81b8cd705de750324e124355cb70f3fdd8ce788dde0b1c4da529" exitCode=0
Oct 30 14:57:03.509416 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.508987    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" event={"ID":"982473c9-6215-4df5-8a3f-6d88ffbac6e4","Type":"ContainerDied","Data":"2d5ddfdf869a81b8cd705de750324e124355cb70f3fdd8ce788dde0b1c4da529"}
Oct 30 14:57:03.509416 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.509377    5064 scope.go:122] "RemoveContainer" containerID="2d5ddfdf869a81b8cd705de750324e124355cb70f3fdd8ce788dde0b1c4da529"
Oct 30 14:57:03.510552 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.510519    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:57:03.510647 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.510592    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:57:03.510647 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:03.510623    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:57:04.517867 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:04.516711    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" event={"ID":"982473c9-6215-4df5-8a3f-6d88ffbac6e4","Type":"ContainerStarted","Data":"a758fc113ef8bbe95d24a11efda9b3595aa005e7a28c4229c53e3ab9e6f391f7"}
Oct 30 14:57:04.603586 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:04.603550    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:57:04.604630 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:04.604600    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 14:57:04.604761 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:04.604664    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 14:57:05.604750 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:05.604715    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 14:57:05.605183 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:05.604773    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 14:57:06.604420 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:06.604373    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 14:57:06.604635 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:06.604441    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 14:57:06.991981 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:06.991945    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:57:07.604087 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:07.604050    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 14:57:07.604268 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:07.604114    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 14:57:08.604469 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:08.604427    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 14:57:08.604895 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:08.604494    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 14:57:09.607759 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:09.607726    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:57:09.609004 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:09.608977    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl"
Oct 30 14:57:29.630247 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:57:29.630205    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:57:30.762747 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:30.762719    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:57:30.765694 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:30.765671    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 14:57:57.202234 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.202201    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"]
Oct 30 14:57:57.206019 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.205996    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.228106 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.228071    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"]
Oct 30 14:57:57.239770 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.239740    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 14:57:57.239770 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.239768    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 14:57:57.239973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.239740    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 14:57:57.239973 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.239819    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9r8cs\""
Oct 30 14:57:57.336402 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336367    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.336585 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336417    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772bv\" (UniqueName: \"kubernetes.io/projected/d20bc29c-6f82-4373-87c2-dd32b724e7dd-kube-api-access-772bv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.336585 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336536    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.336658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336585    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.336691 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336678    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.336724 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.336716    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437622 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437576    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437852 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437639    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437852 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437691    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437852 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437717    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437852 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437759    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.437852 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.437798    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772bv\" (UniqueName: \"kubernetes.io/projected/d20bc29c-6f82-4373-87c2-dd32b724e7dd-kube-api-access-772bv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.438133 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.438109    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.438187 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.438147    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.438231 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.438193    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.440014 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.439989    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20bc29c-6f82-4373-87c2-dd32b724e7dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.440254 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.440233    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20bc29c-6f82-4373-87c2-dd32b724e7dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.463091 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.463016    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772bv\" (UniqueName: \"kubernetes.io/projected/d20bc29c-6f82-4373-87c2-dd32b724e7dd-kube-api-access-772bv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hptln\" (UID: \"d20bc29c-6f82-4373-87c2-dd32b724e7dd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.521907 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.521862    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:57:57.827641 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:57.827506    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"]
Oct 30 14:57:57.830475 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:57:57.830441    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20bc29c_6f82_4373_87c2_dd32b724e7dd.slice/crio-2e2fb883cb1cee1acc8b5c2f26d88460683e2d6e12d8091b1b3141317ea82d25 WatchSource:0}: Error finding container 2e2fb883cb1cee1acc8b5c2f26d88460683e2d6e12d8091b1b3141317ea82d25: Status 404 returned error can't find the container with id 2e2fb883cb1cee1acc8b5c2f26d88460683e2d6e12d8091b1b3141317ea82d25
Oct 30 14:57:58.788820 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:58.788781    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln" event={"ID":"d20bc29c-6f82-4373-87c2-dd32b724e7dd","Type":"ContainerStarted","Data":"2e2fb883cb1cee1acc8b5c2f26d88460683e2d6e12d8091b1b3141317ea82d25"}
Oct 30 14:57:59.632303 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:57:59.632268    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:57:59.794753 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:59.794715    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln" event={"ID":"d20bc29c-6f82-4373-87c2-dd32b724e7dd","Type":"ContainerStarted","Data":"9bb5042e9ac021352ee49603894405edec6f2ec080e99c7804e4d2cd41a70a9d"}
Oct 30 14:57:59.841877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:57:59.841804    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln" podStartSLOduration=1.381419046 podStartE2EDuration="2.841786409s" totalImagesPullingTime="1.460367363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:59.84059792 +0000 UTC m=+629.282488637" watchObservedRunningTime="2026-10-30 14:57:59.841786409 +0000 UTC m=+629.283677117"
Oct 30 14:58:00.522302 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:00.522250    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:58:01.491658 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.491615    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"]
Oct 30 14:58:01.501119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.501091    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.528372 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.528335    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"]
Oct 30 14:58:01.533862 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.532302    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 14:58:01.581210 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581166    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.581383 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581220    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6hk\" (UniqueName: \"kubernetes.io/projected/f009e6e3-5135-48c0-864c-c9672de9b17a-kube-api-access-js6hk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.581383 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581337    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.581383 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581370    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f009e6e3-5135-48c0-864c-c9672de9b17a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.581485 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581456    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.581522 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.581515    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.682879 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.682822    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.683961 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.682924    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.683961 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.682969    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6hk\" (UniqueName: \"kubernetes.io/projected/f009e6e3-5135-48c0-864c-c9672de9b17a-kube-api-access-js6hk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.683961 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.683030    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.683961 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.683055    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f009e6e3-5135-48c0-864c-c9672de9b17a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.683961 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.683111    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.684367 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.684285    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.684537 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.684509    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.685369 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.685344    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.687325 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.687300    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f009e6e3-5135-48c0-864c-c9672de9b17a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.687568 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.687545    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f009e6e3-5135-48c0-864c-c9672de9b17a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.709987 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.709952    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6hk\" (UniqueName: \"kubernetes.io/projected/f009e6e3-5135-48c0-864c-c9672de9b17a-kube-api-access-js6hk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9qz5r\" (UID: \"f009e6e3-5135-48c0-864c-c9672de9b17a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:01.815755 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:01.815664    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:02.121034 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:02.121007    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"]
Oct 30 14:58:02.123090 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:02.123059    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf009e6e3_5135_48c0_864c_c9672de9b17a.slice/crio-67e7c4b389d2ef20b3e3b1cf9671f493eab4238a6ff4998b1c85909a49e7d2ea WatchSource:0}: Error finding container 67e7c4b389d2ef20b3e3b1cf9671f493eab4238a6ff4998b1c85909a49e7d2ea: Status 404 returned error can't find the container with id 67e7c4b389d2ef20b3e3b1cf9671f493eab4238a6ff4998b1c85909a49e7d2ea
Oct 30 14:58:02.809976 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:02.809889    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r" event={"ID":"f009e6e3-5135-48c0-864c-c9672de9b17a","Type":"ContainerStarted","Data":"875c026fc5e6448b7d828c8e45738d61dca0aef51354b8e951fa6a47929ccf21"}
Oct 30 14:58:02.809976 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:02.809941    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r" event={"ID":"f009e6e3-5135-48c0-864c-c9672de9b17a","Type":"ContainerStarted","Data":"67e7c4b389d2ef20b3e3b1cf9671f493eab4238a6ff4998b1c85909a49e7d2ea"}
Oct 30 14:58:02.815822 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:02.815801    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:02.858633 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:02.858579    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r" podStartSLOduration=1.617790327 podStartE2EDuration="1.858563969s" totalImagesPullingTime="240.773642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:02.856823607 +0000 UTC m=+632.298714348" watchObservedRunningTime="2026-10-30 14:58:02.858563969 +0000 UTC m=+632.300454680"
Oct 30 14:58:06.478930 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.478894    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"]
Oct 30 14:58:06.482920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.482890    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.502152 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.502119    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"]
Oct 30 14:58:06.513372 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.513340    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 14:58:06.633297 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633264    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.633297 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633304    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.633524 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633340    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.633524 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633438    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42v6j\" (UniqueName: \"kubernetes.io/projected/034782e3-3b6e-4c3b-852b-ea4b23f06a75-kube-api-access-42v6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.633524 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633495    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.633627 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.633549    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.734962 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.734880    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.734962 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.734934    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735217 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.734967    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735217 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735015    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735217 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735149    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v6j\" (UniqueName: \"kubernetes.io/projected/034782e3-3b6e-4c3b-852b-ea4b23f06a75-kube-api-access-42v6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735434 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735231    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735434 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735385    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735434 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735413    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.735668 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.735647    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.737376 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.737355    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034782e3-3b6e-4c3b-852b-ea4b23f06a75-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.737600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.737581    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034782e3-3b6e-4c3b-852b-ea4b23f06a75-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.762633 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.762597    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v6j\" (UniqueName: \"kubernetes.io/projected/034782e3-3b6e-4c3b-852b-ea4b23f06a75-kube-api-access-42v6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx\" (UID: \"034782e3-3b6e-4c3b-852b-ea4b23f06a75\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:06.799766 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:06.799730    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:07.101270 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.101241    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"]
Oct 30 14:58:07.103159 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:07.103128    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034782e3_3b6e_4c3b_852b_ea4b23f06a75.slice/crio-2818ffe8db95483ac1ade8486c93ffbca5539436895b066f99b35aeff10dbc86 WatchSource:0}: Error finding container 2818ffe8db95483ac1ade8486c93ffbca5539436895b066f99b35aeff10dbc86: Status 404 returned error can't find the container with id 2818ffe8db95483ac1ade8486c93ffbca5539436895b066f99b35aeff10dbc86
Oct 30 14:58:07.522233 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.522195    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:58:07.534917 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.534889    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:58:07.839252 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.839193    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx" event={"ID":"034782e3-3b6e-4c3b-852b-ea4b23f06a75","Type":"ContainerStarted","Data":"a6855b2486f8140dd6276cbcd397b4da8f3e14e787e104d124c46b64b49ccf23"}
Oct 30 14:58:07.839557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.839528    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx" event={"ID":"034782e3-3b6e-4c3b-852b-ea4b23f06a75","Type":"ContainerStarted","Data":"2818ffe8db95483ac1ade8486c93ffbca5539436895b066f99b35aeff10dbc86"}
Oct 30 14:58:07.854856 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.854802    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hptln"
Oct 30 14:58:07.887232 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:07.887177    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx" podStartSLOduration=1.645204359 podStartE2EDuration="1.887164154s" totalImagesPullingTime="241.959795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:07.885779501 +0000 UTC m=+637.327670217" watchObservedRunningTime="2026-10-30 14:58:07.887164154 +0000 UTC m=+637.329054871"
Oct 30 14:58:08.800751 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:08.800708    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:11.816128 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:11.816090    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:11.829300 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:11.829266    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:11.872868 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:11.872821    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r"
Oct 30 14:58:16.800798 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:16.800757    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:16.813363 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:16.813341    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:16.899804 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:16.899775    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx"
Oct 30 14:58:27.626033 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.625997    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:27.629689 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.629669    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.653174 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.653138    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:27.665348 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.665318    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f7hxz\""
Oct 30 14:58:27.665590 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.665571    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 14:58:27.665648 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.665569    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 14:58:27.734541 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.734506    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zp47\" (UniqueName: \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.734747 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.734725    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.835340 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.835298    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.835526 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.835372    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zp47\" (UniqueName: \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.837904 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.837873    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.861044 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.861008    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zp47\" (UniqueName: \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\") pod \"ai-gateway-controller-697b76b867-b45bx\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") " pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:27.943203 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:27.943101    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:28.277679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:28.277654    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:28.280371 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:28.280344    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f256517_55d1_47ec_a0e9_23d004588efc.slice/crio-bacaceb0247cfa0c1a308ae878cfdb4b13a5a3ae349bf38e4fb1e3d7f70dd5d6 WatchSource:0}: Error finding container bacaceb0247cfa0c1a308ae878cfdb4b13a5a3ae349bf38e4fb1e3d7f70dd5d6: Status 404 returned error can't find the container with id bacaceb0247cfa0c1a308ae878cfdb4b13a5a3ae349bf38e4fb1e3d7f70dd5d6
Oct 30 14:58:28.957953 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:28.957914    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" event={"ID":"2f256517-55d1-47ec-a0e9-23d004588efc","Type":"ContainerStarted","Data":"bacaceb0247cfa0c1a308ae878cfdb4b13a5a3ae349bf38e4fb1e3d7f70dd5d6"}
Oct 30 14:58:29.633952 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:58:29.633908    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:58:30.971893 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:30.971855    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" event={"ID":"2f256517-55d1-47ec-a0e9-23d004588efc","Type":"ContainerStarted","Data":"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"}
Oct 30 14:58:30.972310 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:30.971957    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:31.018703 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:31.018645    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" podStartSLOduration=1.7833259780000001 podStartE2EDuration="4.018631337s" totalImagesPullingTime="2.235305359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:31.016699429 +0000 UTC m=+660.458590145" watchObservedRunningTime="2026-10-30 14:58:31.018631337 +0000 UTC m=+660.460522054"
Oct 30 14:58:32.886328 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.886273    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"]
Oct 30 14:58:32.890420 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.890394    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:32.912200 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.912162    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"]
Oct 30 14:58:32.922591 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.922565    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 14:58:32.922809 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.922622    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lz6nw\""
Oct 30 14:58:32.969960 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.969925    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5z4d5"]
Oct 30 14:58:32.973663 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.973639    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:32.983501 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.983478    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c109ff3-12cb-4096-a383-20e111d550f8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:32.983775 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.983749    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjhn\" (UniqueName: \"kubernetes.io/projected/2c109ff3-12cb-4096-a383-20e111d550f8-kube-api-access-7vjhn\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:32.994264 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:32.994240    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5z4d5"]
Oct 30 14:58:33.084689 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.084640    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjhn\" (UniqueName: \"kubernetes.io/projected/2c109ff3-12cb-4096-a383-20e111d550f8-kube-api-access-7vjhn\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:33.084977 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.084735    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vb9\" (UniqueName: \"kubernetes.io/projected/78b17b36-bfcb-4f6e-aba5-995d1012fd99-kube-api-access-f6vb9\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.084977 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.084787    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c109ff3-12cb-4096-a383-20e111d550f8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:33.084977 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.084934    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b17b36-bfcb-4f6e-aba5-995d1012fd99-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.085699 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.085680    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c109ff3-12cb-4096-a383-20e111d550f8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:33.110254 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.110219    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjhn\" (UniqueName: \"kubernetes.io/projected/2c109ff3-12cb-4096-a383-20e111d550f8-kube-api-access-7vjhn\") pod \"payload-pre-processing-7b98bb678c-fwvxh\" (UID: \"2c109ff3-12cb-4096-a383-20e111d550f8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:33.185717 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.185628    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vb9\" (UniqueName: \"kubernetes.io/projected/78b17b36-bfcb-4f6e-aba5-995d1012fd99-kube-api-access-f6vb9\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.185717 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.185696    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b17b36-bfcb-4f6e-aba5-995d1012fd99-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.186398 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.186379    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b17b36-bfcb-4f6e-aba5-995d1012fd99-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.206614 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.206579    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:33.211723 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.211698    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vb9\" (UniqueName: \"kubernetes.io/projected/78b17b36-bfcb-4f6e-aba5-995d1012fd99-kube-api-access-f6vb9\") pod \"payload-processing-5f78687d5b-5z4d5\" (UID: \"78b17b36-bfcb-4f6e-aba5-995d1012fd99\") " pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.291945 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.291906    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:33.511752 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.511727    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"]
Oct 30 14:58:33.513925 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:33.513893    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c109ff3_12cb_4096_a383_20e111d550f8.slice/crio-837e737d869217a4cb14e669354f6bf014c51df265c9bb8f50ddb4f37f7d1c34 WatchSource:0}: Error finding container 837e737d869217a4cb14e669354f6bf014c51df265c9bb8f50ddb4f37f7d1c34: Status 404 returned error can't find the container with id 837e737d869217a4cb14e669354f6bf014c51df265c9bb8f50ddb4f37f7d1c34
Oct 30 14:58:33.598530 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.598505    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5z4d5"]
Oct 30 14:58:33.600758 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:33.600727    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b17b36_bfcb_4f6e_aba5_995d1012fd99.slice/crio-82d1bdd9351a8a38f6d540589381678e07624a18d03161992a293ea093676c2c WatchSource:0}: Error finding container 82d1bdd9351a8a38f6d540589381678e07624a18d03161992a293ea093676c2c: Status 404 returned error can't find the container with id 82d1bdd9351a8a38f6d540589381678e07624a18d03161992a293ea093676c2c
Oct 30 14:58:33.986761 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.986723    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5" event={"ID":"78b17b36-bfcb-4f6e-aba5-995d1012fd99","Type":"ContainerStarted","Data":"82d1bdd9351a8a38f6d540589381678e07624a18d03161992a293ea093676c2c"}
Oct 30 14:58:33.987949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:33.987925    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh" event={"ID":"2c109ff3-12cb-4096-a383-20e111d550f8","Type":"ContainerStarted","Data":"837e737d869217a4cb14e669354f6bf014c51df265c9bb8f50ddb4f37f7d1c34"}
Oct 30 14:58:35.405573 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.405534    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 14:58:35.420798 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.420769    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.442789 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.441802    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 14:58:35.509323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509280    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509517 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509351    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509522    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509649 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509584    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509712 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509656    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509712 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509699    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509819 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509800    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509899 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509864    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.509960 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.509896    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlgb\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610605    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610648    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610669    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlgb\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610711    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610745    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610773    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610798    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.610869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610823    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611211 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.610879    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611211 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.611168    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611325 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.611300    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611384 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.611358    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611548 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.611524    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.611905 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.611882    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.613754 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.613710    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.613877 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.613817    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.636680 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.636649    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.637007 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.636984    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlgb\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\") pod \"maas-default-gateway-openshift-default-6cc458d887-g2bcr\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:35.744517 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:35.744427    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:36.078976 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:36.078951    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 14:58:36.081071 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:36.081034    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c1af33_79ab_4202_b9e8_000ab7e22184.slice/crio-1d565de192dffe86937f098287e2b6e84e921d3f2c16743ae11156049cf6ce62 WatchSource:0}: Error finding container 1d565de192dffe86937f098287e2b6e84e921d3f2c16743ae11156049cf6ce62: Status 404 returned error can't find the container with id 1d565de192dffe86937f098287e2b6e84e921d3f2c16743ae11156049cf6ce62
Oct 30 14:58:36.083290 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:36.083255    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:58:36.083359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:36.083321    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:58:36.083359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:36.083349    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:58:37.006151 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.006116    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5" event={"ID":"78b17b36-bfcb-4f6e-aba5-995d1012fd99","Type":"ContainerStarted","Data":"693bb92de8f021a6d3237f1b987a1e39267bccb62e7dd940409d0bd3b7113f95"}
Oct 30 14:58:37.006583 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.006242    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:37.007782 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.007748    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh" event={"ID":"2c109ff3-12cb-4096-a383-20e111d550f8","Type":"ContainerStarted","Data":"660abac785e932d5271acd608a07583c60cc0edf453a8bcf8e788af15b42f2f7"}
Oct 30 14:58:37.007934 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.007805    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:37.009306 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.009283    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" event={"ID":"30c1af33-79ab-4202-b9e8-000ab7e22184","Type":"ContainerStarted","Data":"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"}
Oct 30 14:58:37.009424 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.009319    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" event={"ID":"30c1af33-79ab-4202-b9e8-000ab7e22184","Type":"ContainerStarted","Data":"1d565de192dffe86937f098287e2b6e84e921d3f2c16743ae11156049cf6ce62"}
Oct 30 14:58:37.061378 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.061325    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5" podStartSLOduration=2.697694651 podStartE2EDuration="5.061312292s" totalImagesPullingTime="2.363617641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:37.060559532 +0000 UTC m=+666.502450250" watchObservedRunningTime="2026-10-30 14:58:37.061312292 +0000 UTC m=+666.503203006"
Oct 30 14:58:37.110591 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.110538    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podStartSLOduration=2.110524565 podStartE2EDuration="2.110524565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:37.108941196 +0000 UTC m=+666.550831924" watchObservedRunningTime="2026-10-30 14:58:37.110524565 +0000 UTC m=+666.552415282"
Oct 30 14:58:37.166865 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.166791    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh" podStartSLOduration=2.74680855 podStartE2EDuration="5.166775584s" totalImagesPullingTime="2.419967034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:37.166110206 +0000 UTC m=+666.608000923" watchObservedRunningTime="2026-10-30 14:58:37.166775584 +0000 UTC m=+666.608666301"
Oct 30 14:58:37.745169 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.745123    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:37.746875 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.746851    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:37.746971 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:37.746904    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:38.744886 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:38.744820    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:38.745386 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:38.744916    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:39.745557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:39.745511    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:39.746105 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:39.745602    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:40.745179 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:40.745146    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:40.745363 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:40.745209    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:41.745408 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:41.745367    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:41.745863 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:41.745457    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:41.980160 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:41.980123    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:42.294336 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.294303    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"]
Oct 30 14:58:42.299435 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.299408    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.316707 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.316678    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"]
Oct 30 14:58:42.394367 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.394331    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc8a5aa-fcda-4035-88cb-de3efce8b081-metrics-certs\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.394522 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.394415    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lb5s\" (UniqueName: \"kubernetes.io/projected/cdc8a5aa-fcda-4035-88cb-de3efce8b081-kube-api-access-9lb5s\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.495496 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.495454    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lb5s\" (UniqueName: \"kubernetes.io/projected/cdc8a5aa-fcda-4035-88cb-de3efce8b081-kube-api-access-9lb5s\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.495691 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.495547    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc8a5aa-fcda-4035-88cb-de3efce8b081-metrics-certs\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.498226 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.498193    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc8a5aa-fcda-4035-88cb-de3efce8b081-metrics-certs\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.521689 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.521649    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lb5s\" (UniqueName: \"kubernetes.io/projected/cdc8a5aa-fcda-4035-88cb-de3efce8b081-kube-api-access-9lb5s\") pod \"ai-gateway-controller-65cd8c9c54-sqsrv\" (UID: \"cdc8a5aa-fcda-4035-88cb-de3efce8b081\") " pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.613068 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.612977    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:42.745162 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.745116    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:58:42.745367 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.745172    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 14:58:42.918864 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:42.918755    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"]
Oct 30 14:58:42.921904 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:58:42.921871    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc8a5aa_fcda_4035_88cb_de3efce8b081.slice/crio-bbc4300db13fa3b816fed3c6b5610804ca2d88124b52727a944ecbf55d527b4d WatchSource:0}: Error finding container bbc4300db13fa3b816fed3c6b5610804ca2d88124b52727a944ecbf55d527b4d: Status 404 returned error can't find the container with id bbc4300db13fa3b816fed3c6b5610804ca2d88124b52727a944ecbf55d527b4d
Oct 30 14:58:43.062060 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:43.062023    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv" event={"ID":"cdc8a5aa-fcda-4035-88cb-de3efce8b081","Type":"ContainerStarted","Data":"bbc4300db13fa3b816fed3c6b5610804ca2d88124b52727a944ecbf55d527b4d"}
Oct 30 14:58:44.068609 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.068570    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv" event={"ID":"cdc8a5aa-fcda-4035-88cb-de3efce8b081","Type":"ContainerStarted","Data":"3c8665e03eea047591fdd13f865450f14bc809f5e6da5665bc8ab9e633367143"}
Oct 30 14:58:44.069069 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.068752    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:44.117839 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.117767    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv" podStartSLOduration=1.83570867 podStartE2EDuration="2.117747855s" totalImagesPullingTime="282.039185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:44.114511737 +0000 UTC m=+673.556402520" watchObservedRunningTime="2026-10-30 14:58:44.117747855 +0000 UTC m=+673.559638572"
Oct 30 14:58:44.467813 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.467780    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:44.468309 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.468282    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:44.469783 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:44.469758    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 14:58:48.018675 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:48.018644    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-5z4d5"
Oct 30 14:58:48.019104 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:48.018994    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-fwvxh"
Oct 30 14:58:55.076706 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.076676    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-65cd8c9c54-sqsrv"
Oct 30 14:58:55.150187 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.150152    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:55.150462 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.150428    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" podUID="2f256517-55d1-47ec-a0e9-23d004588efc" containerName="manager" containerID="cri-o://102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42" gracePeriod=30
Oct 30 14:58:55.418063 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.418038    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:55.526700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.526662    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\" (UniqueName: \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\") pod \"2f256517-55d1-47ec-a0e9-23d004588efc\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") "
Oct 30 14:58:55.526700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.526699    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\") pod \"2f256517-55d1-47ec-a0e9-23d004588efc\" (UID: \"2f256517-55d1-47ec-a0e9-23d004588efc\") "
Oct 30 14:58:55.528888 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.528855    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47" pod "2f256517-55d1-47ec-a0e9-23d004588efc" (UID: "2f256517-55d1-47ec-a0e9-23d004588efc"). InnerVolumeSpecName "kube-api-access-6zp47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:58:55.529027 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.528950    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs" pod "2f256517-55d1-47ec-a0e9-23d004588efc" (UID: "2f256517-55d1-47ec-a0e9-23d004588efc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:58:55.627636 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.627598    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zp47\" (UniqueName: \"kubernetes.io/projected/2f256517-55d1-47ec-a0e9-23d004588efc-kube-api-access-6zp47\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:58:55.627636 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:55.627630    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f256517-55d1-47ec-a0e9-23d004588efc-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 14:58:56.130774 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.130737    5064 generic.go:356] "Generic (PLEG): container finished" podID="2f256517-55d1-47ec-a0e9-23d004588efc" containerID="102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42" exitCode=0
Oct 30 14:58:56.131282 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.130779    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx"
Oct 30 14:58:56.131282 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.130797    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" event={"ID":"2f256517-55d1-47ec-a0e9-23d004588efc","Type":"ContainerDied","Data":"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"}
Oct 30 14:58:56.131282 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.130866    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-b45bx" event={"ID":"2f256517-55d1-47ec-a0e9-23d004588efc","Type":"ContainerDied","Data":"bacaceb0247cfa0c1a308ae878cfdb4b13a5a3ae349bf38e4fb1e3d7f70dd5d6"}
Oct 30 14:58:56.131282 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.130902    5064 scope.go:122] "RemoveContainer" containerID="102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"
Oct 30 14:58:56.140848 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.140821    5064 scope.go:122] "RemoveContainer" containerID="102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"
Oct 30 14:58:56.141126 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:58:56.141108    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42\": container with ID starting with 102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42 not found: ID does not exist" containerID="102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"
Oct 30 14:58:56.141172 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.141136    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42"} err="failed to get container status \"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42\": rpc error: code = NotFound desc = could not find container \"102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42\": container with ID starting with 102e41d55806dc19ee35ef5a7830f894d325d9d712f58793e40aeea14f29ef42 not found: ID does not exist"
Oct 30 14:58:56.188048 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.188009    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:56.202043 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.202013    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-b45bx"]
Oct 30 14:58:56.721952 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:58:56.721917    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f256517-55d1-47ec-a0e9-23d004588efc" path="/var/lib/kubelet/pods/2f256517-55d1-47ec-a0e9-23d004588efc/volumes"
Oct 30 14:58:59.630576 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:58:59.630538    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:59:22.988374 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:22.988340    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 14:59:22.989044 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:22.989022    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2f256517-55d1-47ec-a0e9-23d004588efc" containerName="manager"
Oct 30 14:59:22.989130 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:22.989045    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f256517-55d1-47ec-a0e9-23d004588efc" containerName="manager"
Oct 30 14:59:22.989220 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:22.989206    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f256517-55d1-47ec-a0e9-23d004588efc" containerName="manager"
Oct 30 14:59:22.993076 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:22.993045    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.017492 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.017467    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e64e05-openshift-default-dockercfg-s2txv\""
Oct 30 14:59:23.031814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.031782    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 14:59:23.089422 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089388    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089422 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089426    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089447    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4w4\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089565    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089619 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089591    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089732 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089628    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089732 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089691    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089732 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089726    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.089858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.089745    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.170753 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.170715    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 14:59:23.174957 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.174932    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.190410 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190385    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190416    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190445    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190481    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190511    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190557 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190534    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190628    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190660    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.190885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190806    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4w4\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.191036 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190905    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.191036 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.190955    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.191137 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.191109    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.191401 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.191380    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.191639 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.191624    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.192753 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.192729    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.193267 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.193243    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.195188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.195169    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-4f5c30-openshift-default-dockercfg-pvckr\""
Oct 30 14:59:23.211094 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.211069    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 14:59:23.221323 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.221297    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4w4\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.224869 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.224845    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.292416 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292322    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292416 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292370    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292416 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292398    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292561    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292598    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292733 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292641    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292900 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292846    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.292962 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.292901    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.293198 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.293013    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pj4f\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.310131 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.310090    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:23.394872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394723    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.394872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394763    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.394872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394787    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.394872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394815    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.394872 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394869    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394894    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pj4f\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394954    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.394979    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395262 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.395014    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395472 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.395440    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395645 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.395627    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.395800 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.395784    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.397007 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.396948    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.397421 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.397365    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.397817 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.397799    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.399235 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.399211    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.425726 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.425682    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.425966 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.425939    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pj4f\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.491305 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.491111    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:23.501422 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.501386    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 14:59:23.505999 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.505979    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.527170 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.527118    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2c8d89-openshift-default-dockercfg-mxn4h\""
Oct 30 14:59:23.528757 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.528715    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597662    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597738    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597774    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597850    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597892    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597930    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5fp\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.597962    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.598005    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.598223 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.598040    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.640587 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.640562    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 14:59:23.642519 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:23.642487    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-35b87760b6a1cac67e73dc9e3adca1a9347fa26e506ef79736e062f0cc00663d WatchSource:0}: Error finding container 35b87760b6a1cac67e73dc9e3adca1a9347fa26e506ef79736e062f0cc00663d: Status 404 returned error can't find the container with id 35b87760b6a1cac67e73dc9e3adca1a9347fa26e506ef79736e062f0cc00663d
Oct 30 14:59:23.644655 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.644618    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:23.644743 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.644690    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:23.644743 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.644720    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:23.699447 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699410    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699608 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699559    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699647    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699670    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699786 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699722    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699786 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699754    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699926 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699786    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5fp\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699926 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699807    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.699926 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.699858    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.700069 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.700052    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.700279 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.700252    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.700348 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.700334    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.700516 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.700494    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.700885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.700864    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.702205 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.702184    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.702462 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.702444    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.733157 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.733119    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5fp\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.733393 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.733372    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.824296 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.824054    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:23.825077 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.825032    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 14:59:23.827406 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:23.827382    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9905b36f_8be0_460c_abc4_acd7404396ba.slice/crio-a5fc40c4025429ff98dc247c981e0cc1ab16f5d3b6a4bf96b60d6a25aee8ee17 WatchSource:0}: Error finding container a5fc40c4025429ff98dc247c981e0cc1ab16f5d3b6a4bf96b60d6a25aee8ee17: Status 404 returned error can't find the container with id a5fc40c4025429ff98dc247c981e0cc1ab16f5d3b6a4bf96b60d6a25aee8ee17
Oct 30 14:59:23.835652 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.835576    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:23.835800 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.835677    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:23.835800 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:23.835720    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.128405 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.128373    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 14:59:24.133011 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.132976    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.155308 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.155279    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-83ff4e-openshift-default-dockercfg-l8gp9\""
Oct 30 14:59:24.165275 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.164975    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 14:59:24.179478 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.179445    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 14:59:24.184192 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:24.184158    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28085bb2_5d6a_43fe_b036_af6921999478.slice/crio-1c20e629274b7e90842f6d6ab9b585434037e0ee36fa17f8bca923e125d49885 WatchSource:0}: Error finding container 1c20e629274b7e90842f6d6ab9b585434037e0ee36fa17f8bca923e125d49885: Status 404 returned error can't find the container with id 1c20e629274b7e90842f6d6ab9b585434037e0ee36fa17f8bca923e125d49885
Oct 30 14:59:24.187250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.187183    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.187465 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.187431    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.187573 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.187495    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.211148 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211119    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211280 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211169    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211280 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211274    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211361 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211306    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211414 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211388    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211458 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211436    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211503 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211471    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmhh\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211575 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211564    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.211620 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.211583    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.268112 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.268077    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerStarted","Data":"c1da23b380d51178335418ca0679a2bafb26b00ca9df356a11af454b12d8b253"}
Oct 30 14:59:24.268272 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.268143    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerStarted","Data":"35b87760b6a1cac67e73dc9e3adca1a9347fa26e506ef79736e062f0cc00663d"}
Oct 30 14:59:24.270086 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.270048    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerStarted","Data":"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"}
Oct 30 14:59:24.270225 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.270102    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerStarted","Data":"a5fc40c4025429ff98dc247c981e0cc1ab16f5d3b6a4bf96b60d6a25aee8ee17"}
Oct 30 14:59:24.272419 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.272396    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerStarted","Data":"1c20e629274b7e90842f6d6ab9b585434037e0ee36fa17f8bca923e125d49885"}
Oct 30 14:59:24.310387 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.310350    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:24.312196 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312168    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312196 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312198    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312396 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312251    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312396 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312341    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312409    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312570 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312509    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312626 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312578    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312626 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312583    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312626 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312604    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312780 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312636    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.312780 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.312663    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmhh\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.315446 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.314745    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.315698 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.315677    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.316095 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.315961    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.316095 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.316060    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.316719 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.316699    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:24.316881 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.316863    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.356736 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.356699    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.362957 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.362592    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmhh\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.399608 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.399505    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podStartSLOduration=2.399491367 podStartE2EDuration="2.399491367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:24.347348073 +0000 UTC m=+713.789238803" watchObservedRunningTime="2026-10-30 14:59:24.399491367 +0000 UTC m=+713.841382084"
Oct 30 14:59:24.399778 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.399756    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podStartSLOduration=1.399750931 podStartE2EDuration="1.399750931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:24.397657486 +0000 UTC m=+713.839548205" watchObservedRunningTime="2026-10-30 14:59:24.399750931 +0000 UTC m=+713.841641648"
Oct 30 14:59:24.452236 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.452200    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:24.491938 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.491876    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:24.499521 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.499485    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:24.795954 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.795928    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 14:59:24.798304 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:24.798272    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec3b842_e8c3_42fe_a24f_0c3def05b372.slice/crio-908b605dae501251bf417e4b394c6c6b4d778a35e5a4f2243da743018c1a41b3 WatchSource:0}: Error finding container 908b605dae501251bf417e4b394c6c6b4d778a35e5a4f2243da743018c1a41b3: Status 404 returned error can't find the container with id 908b605dae501251bf417e4b394c6c6b4d778a35e5a4f2243da743018c1a41b3
Oct 30 14:59:24.800683 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.800647    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.800797 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.800729    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:24.800797 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:24.800773    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:25.286793 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.286748    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerStarted","Data":"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"}
Oct 30 14:59:25.288613 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.288580    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerStarted","Data":"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"}
Oct 30 14:59:25.288770 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.288626    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerStarted","Data":"908b605dae501251bf417e4b394c6c6b4d778a35e5a4f2243da743018c1a41b3"}
Oct 30 14:59:25.289127 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.289095    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:25.289257 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.289142    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:25.290121 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.290093    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:25.290227 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.290208    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:25.342000 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.341947    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podStartSLOduration=2.341933395 podStartE2EDuration="2.341933395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:25.34011628 +0000 UTC m=+714.782006998" watchObservedRunningTime="2026-10-30 14:59:25.341933395 +0000 UTC m=+714.783824111"
Oct 30 14:59:25.453703 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.453667    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:25.459039 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.459006    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:25.501370 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.501315    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podStartSLOduration=1.501300332 podStartE2EDuration="1.501300332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:25.499789282 +0000 UTC m=+714.941679999" watchObservedRunningTime="2026-10-30 14:59:25.501300332 +0000 UTC m=+714.943191048"
Oct 30 14:59:25.824566 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.824526    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:25.830348 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:25.830319    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:26.294721 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:26.294688    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:26.295210 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:26.294746    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:26.295277 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:26.295253    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 14:59:26.295623 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:26.295603    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 14:59:28.730222 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.730188    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 14:59:28.736059 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.736032    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.760916 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.760886    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 14:59:28.761173 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.761154    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e64e05-serving-cert\""
Oct 30 14:59:28.868711 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.868675    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.868941 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.868891    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.869007 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.868944    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfcv\" (UniqueName: \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.869059 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.869002    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.970241 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.970200    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.970419 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.970257    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfcv\" (UniqueName: \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.970419 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.970359    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.970570 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.970546    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.970952 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.970932    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.972941 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.972908    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:28.973091 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:28.972996    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:29.002106 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:29.002032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfcv\" (UniqueName: \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\") pod \"maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:29.051088 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:29.051047    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:29.369618 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:29.369576    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 14:59:29.376492 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:29.376456    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ec3a0_dfc5_4756_870c_9199ce37a0d0.slice/crio-1993449582af4f63871f530437c19cd906ebca6d03c55ec25206d7da9eeb2992 WatchSource:0}: Error finding container 1993449582af4f63871f530437c19cd906ebca6d03c55ec25206d7da9eeb2992: Status 404 returned error can't find the container with id 1993449582af4f63871f530437c19cd906ebca6d03c55ec25206d7da9eeb2992
Oct 30 14:59:29.630576 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:29.630445    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 14:59:30.320822 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.320738    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" event={"ID":"a07ec3a0-dfc5-4756-870c-9199ce37a0d0","Type":"ContainerStarted","Data":"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"}
Oct 30 14:59:30.321211 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.321010    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:30.321211 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.321051    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" event={"ID":"a07ec3a0-dfc5-4756-870c-9199ce37a0d0","Type":"ContainerStarted","Data":"1993449582af4f63871f530437c19cd906ebca6d03c55ec25206d7da9eeb2992"}
Oct 30 14:59:30.374709 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.374649    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" podStartSLOduration=1.886489638 podStartE2EDuration="2.374635027s" totalImagesPullingTime="488.145389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:30.371595858 +0000 UTC m=+719.813486576" watchObservedRunningTime="2026-10-30 14:59:30.374635027 +0000 UTC m=+719.816525744"
Oct 30 14:59:30.984233 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.984203    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 14:59:30.989411 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:30.989390    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.012848 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.012806    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 14:59:31.013015 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.012978    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-4f5c30\""
Oct 30 14:59:31.023861 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.023792    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-4f5c30-dockercfg-47svg\""
Oct 30 14:59:31.142312 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.142279    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5r47\" (UniqueName: \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.142491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.142333    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.243979 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.243932    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.244189 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.244135    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5r47\" (UniqueName: \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.244741 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.244716    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.273249 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.273226    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5r47\" (UniqueName: \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\") pod \"payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.301956 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.301930    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:31.321367 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.321341    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 14:59:31.325779 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.325757    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.345236 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.345202    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.345421 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.345262    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpg7\" (UniqueName: \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.345421 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.345377    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.345544 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.345424    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.349048 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.349022    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-4f5c30-serving-cert\""
Oct 30 14:59:31.352334 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.352311    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 14:59:31.446908 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.446792    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.446908 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.446854    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.447148 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:31.446957    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-serving-cert: secret "maas-api-e2e-worker-w5-4f5c30-serving-cert" not found
Oct 30 14:59:31.447148 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.446983    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.447148 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.447012    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpg7\" (UniqueName: \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.447148 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:31.447027    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls podName:8f7bef0f-cc45-4218-81cb-838abf3f111e nodeName:}" failed. No retries permitted until 2026-10-30 14:59:31.947011392 +0000 UTC m=+721.388902087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls") pod "maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" (UID: "8f7bef0f-cc45-4218-81cb-838abf3f111e") : secret "maas-api-e2e-worker-w5-4f5c30-serving-cert" not found
Oct 30 14:59:31.447679 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.447653    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.449682 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.449656    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.493948 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.493914    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpg7\" (UniqueName: \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.629644 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.629603    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 14:59:31.634632 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:31.634596    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631c3edf_e93b_484d_ac7a_6dc6f91d8505.slice/crio-b6352ccefea18d711350b9d58eee36274359a4f3bb1583b89b7057496d96c684 WatchSource:0}: Error finding container b6352ccefea18d711350b9d58eee36274359a4f3bb1583b89b7057496d96c684: Status 404 returned error can't find the container with id b6352ccefea18d711350b9d58eee36274359a4f3bb1583b89b7057496d96c684
Oct 30 14:59:31.953887 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.953777    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:31.956391 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:31.956364    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:32.242942 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.242856    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:32.334157 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.334113    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" event={"ID":"631c3edf-e93b-484d-ac7a-6dc6f91d8505","Type":"ContainerStarted","Data":"c6fee53f30bb72e2a29ee8b754841b08e93ae5dee1acb8785be3a27b0cb5ec47"}
Oct 30 14:59:32.334920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.334176    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" event={"ID":"631c3edf-e93b-484d-ac7a-6dc6f91d8505","Type":"ContainerStarted","Data":"b6352ccefea18d711350b9d58eee36274359a4f3bb1583b89b7057496d96c684"}
Oct 30 14:59:32.334920 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.334224    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:32.379243 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.379185    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" podStartSLOduration=2.379171575 podStartE2EDuration="2.379171575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:32.378687094 +0000 UTC m=+721.820577815" watchObservedRunningTime="2026-10-30 14:59:32.379171575 +0000 UTC m=+721.821062308"
Oct 30 14:59:32.559281 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.559247    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 14:59:32.560038 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:32.559995    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7bef0f_cc45_4218_81cb_838abf3f111e.slice/crio-c63f6541e38cfae4e3950c8c0e89f9c9b1ad4c7f4bc0c0115239542d6cf3b642 WatchSource:0}: Error finding container c63f6541e38cfae4e3950c8c0e89f9c9b1ad4c7f4bc0c0115239542d6cf3b642: Status 404 returned error can't find the container with id c63f6541e38cfae4e3950c8c0e89f9c9b1ad4c7f4bc0c0115239542d6cf3b642
Oct 30 14:59:32.561588 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:32.561564    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:59:33.341335 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:33.341243    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" event={"ID":"8f7bef0f-cc45-4218-81cb-838abf3f111e","Type":"ContainerStarted","Data":"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"}
Oct 30 14:59:33.341335 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:33.341330    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" event={"ID":"8f7bef0f-cc45-4218-81cb-838abf3f111e","Type":"ContainerStarted","Data":"c63f6541e38cfae4e3950c8c0e89f9c9b1ad4c7f4bc0c0115239542d6cf3b642"}
Oct 30 14:59:33.341796 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:33.341486    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:33.391862 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:33.391122    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" podStartSLOduration=2.010016443 podStartE2EDuration="2.391108782s" totalImagesPullingTime="381.092339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:33.389642624 +0000 UTC m=+722.831533341" watchObservedRunningTime="2026-10-30 14:59:33.391108782 +0000 UTC m=+722.832999498"
Oct 30 14:59:34.504190 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.504155    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 14:59:34.508444 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.508416    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.546948 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.546887    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 14:59:34.547116 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.547056    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-05d396\""
Oct 30 14:59:34.547210 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.547177    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-05d396-dockercfg-sqb7c\""
Oct 30 14:59:34.597380 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.597337    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 14:59:34.600824 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.600767    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vfc\" (UniqueName: \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.601188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.601163    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.606157 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.606118    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.656590 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.656555    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 14:59:34.702382 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.702348    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqgc\" (UniqueName: \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.702544 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.702477    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.702610 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.702551    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.702610 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.702602    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vfc\" (UniqueName: \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.703296 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.703275    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.744010 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.743970    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vfc\" (UniqueName: \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\") pod \"payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.803810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.803703    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.804033 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.803861    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqgc\" (UniqueName: \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.804490 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.804471    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.839014 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.838973    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqgc\" (UniqueName: \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\") pod \"payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:34.849700 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.849667    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:34.969256 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:34.969217    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:35.154307 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.154267    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 14:59:35.155869 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:35.155819    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b507ad_1592_40a9_88e8_cf4ecb3a6698.slice/crio-eac4276ce24db6cdc11d72974852c3193d5d1540f8cabc40c485d96548753883 WatchSource:0}: Error finding container eac4276ce24db6cdc11d72974852c3193d5d1540f8cabc40c485d96548753883: Status 404 returned error can't find the container with id eac4276ce24db6cdc11d72974852c3193d5d1540f8cabc40c485d96548753883
Oct 30 14:59:35.343393 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.343356    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 14:59:35.355960 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.355926    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" event={"ID":"d274a787-7d52-46c4-ab60-e1db2d53b04f","Type":"ContainerStarted","Data":"8d51242db7a5d3b1df334a98c5c0ba93f52adb8380b717a5af195ba47de421db"}
Oct 30 14:59:35.358285 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.358254    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" event={"ID":"d2b507ad-1592-40a9-88e8-cf4ecb3a6698","Type":"ContainerStarted","Data":"d202c166d79afd62bded588c90157102a7dddc9d26d5bc3177fb2350d54e4f66"}
Oct 30 14:59:35.358395 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.358314    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" event={"ID":"d2b507ad-1592-40a9-88e8-cf4ecb3a6698","Type":"ContainerStarted","Data":"eac4276ce24db6cdc11d72974852c3193d5d1540f8cabc40c485d96548753883"}
Oct 30 14:59:35.359202 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.359181    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:35.388540 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.388511    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 14:59:35.394143 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.394118    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.411351 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.410940    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.411351 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.411038    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mltw\" (UniqueName: \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.411542 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.411407    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" podStartSLOduration=1.411394249 podStartE2EDuration="1.411394249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:35.410386164 +0000 UTC m=+724.852276880" watchObservedRunningTime="2026-10-30 14:59:35.411394249 +0000 UTC m=+724.853284957"
Oct 30 14:59:35.424940 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.417068    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-2c8d89\""
Oct 30 14:59:35.445082 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.437772    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-2c8d89-dockercfg-dktlb\""
Oct 30 14:59:35.448199 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.448145    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 14:59:35.512150 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.512112    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.512628 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.512193    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mltw\" (UniqueName: \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.533855 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.522076    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.567899 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.562560    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mltw\" (UniqueName: \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\") pod \"payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:35.754453 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:35.754416    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:36.101710 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.101681    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 14:59:36.103554 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:36.103523    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afd3f97_5981_4624_91b5_519fa4b031a8.slice/crio-77654b67d7170278a1e7ac8a9e7db2bdac5afadd25ffae7d0271bae885b0bd71 WatchSource:0}: Error finding container 77654b67d7170278a1e7ac8a9e7db2bdac5afadd25ffae7d0271bae885b0bd71: Status 404 returned error can't find the container with id 77654b67d7170278a1e7ac8a9e7db2bdac5afadd25ffae7d0271bae885b0bd71
Oct 30 14:59:36.347196 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.346815    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 14:59:36.365500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.365412    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" event={"ID":"6afd3f97-5981-4624-91b5-519fa4b031a8","Type":"ContainerStarted","Data":"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"}
Oct 30 14:59:36.365500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.365477    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" event={"ID":"6afd3f97-5981-4624-91b5-519fa4b031a8","Type":"ContainerStarted","Data":"77654b67d7170278a1e7ac8a9e7db2bdac5afadd25ffae7d0271bae885b0bd71"}
Oct 30 14:59:36.366273 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.366251    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:36.372087 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.369842    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" event={"ID":"d274a787-7d52-46c4-ab60-e1db2d53b04f","Type":"ContainerStarted","Data":"581b5fe06683251667dd9d7dd97dfc357ebb9a6888961b8999963d6bfcc3d5c5"}
Oct 30 14:59:36.372087 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.369900    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:36.488982 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.488876    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" podStartSLOduration=2.488860458 podStartE2EDuration="2.488860458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:36.44265418 +0000 UTC m=+725.884544898" watchObservedRunningTime="2026-10-30 14:59:36.488860458 +0000 UTC m=+725.930751175"
Oct 30 14:59:36.489298 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:36.489273    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" podStartSLOduration=1.489267817 podStartE2EDuration="1.489267817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:36.487573146 +0000 UTC m=+725.929463863" watchObservedRunningTime="2026-10-30 14:59:36.489267817 +0000 UTC m=+725.931158534"
Oct 30 14:59:37.850354 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.849821    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 14:59:37.855243 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.855213    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.876543 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.876502    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.876882 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.876822    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.877033 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.877018    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmsj\" (UniqueName: \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.877250 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.877235    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.879589 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.879564    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 14:59:37.889061 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.889034    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-05d396-serving-cert\""
Oct 30 14:59:37.978876 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.978825    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.978876 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.978896    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.979165 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.978918    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmsj\" (UniqueName: \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.979165 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.979019    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.979539 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:37.979507    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-serving-cert: secret "maas-api-e2e-worker-w0-05d396-serving-cert" not found
Oct 30 14:59:37.979634 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:37.979579    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls podName:bd778804-48d0-4ef0-9b62-683b1ba8f171 nodeName:}" failed. No retries permitted until 2026-10-30 14:59:38.479559442 +0000 UTC m=+727.921450154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls") pod "maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" (UID: "bd778804-48d0-4ef0-9b62-683b1ba8f171") : secret "maas-api-e2e-worker-w0-05d396-serving-cert" not found
Oct 30 14:59:37.979634 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.979593    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:37.981600 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:37.981578    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:38.064182 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:38.064148    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmsj\" (UniqueName: \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:38.490984 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:38.490938    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:38.493773 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:38.493740    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") pod \"maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:38.772123 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:38.772028    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:39.166564 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:39.166516    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 14:59:39.169403 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:39.169356    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd778804_48d0_4ef0_9b62_683b1ba8f171.slice/crio-c9b3a6ea8eee68110a8fd9eb6cafd01fcae77fb68820fdb69907f12b99c00ccd WatchSource:0}: Error finding container c9b3a6ea8eee68110a8fd9eb6cafd01fcae77fb68820fdb69907f12b99c00ccd: Status 404 returned error can't find the container with id c9b3a6ea8eee68110a8fd9eb6cafd01fcae77fb68820fdb69907f12b99c00ccd
Oct 30 14:59:39.367874 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:39.367757    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 14:59:39.392731 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:39.392642    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" event={"ID":"bd778804-48d0-4ef0-9b62-683b1ba8f171","Type":"ContainerStarted","Data":"c9b3a6ea8eee68110a8fd9eb6cafd01fcae77fb68820fdb69907f12b99c00ccd"}
Oct 30 14:59:40.051845 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.051801    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 14:59:40.055709 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.055691    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.089865 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.082461    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 14:59:40.089865 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.083312    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-357948-serving-cert\""
Oct 30 14:59:40.156017 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.155972    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.156188 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.156154    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bc8l\" (UniqueName: \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.156289 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.156222    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.156340 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.156330    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.257441 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.257390    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.257913 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.257455    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.257913 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.257541    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.257913 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.257675    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc8l\" (UniqueName: \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.258206 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.258177    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.260175 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.260149    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.260612 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.260590    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.283639 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.283607    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bc8l\" (UniqueName: \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\") pod \"maas-api-e2e-worker-w3-357948-845c9476b7-fb97k\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.373005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.372963    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:40.401329 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.401295    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" event={"ID":"bd778804-48d0-4ef0-9b62-683b1ba8f171","Type":"ContainerStarted","Data":"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"}
Oct 30 14:59:40.402166 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.402140    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:40.458946 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.458870    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" podStartSLOduration=3.025801613 podStartE2EDuration="3.458850583s" totalImagesPullingTime="433.04897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:40.457203036 +0000 UTC m=+729.899093768" watchObservedRunningTime="2026-10-30 14:59:40.458850583 +0000 UTC m=+729.900741297"
Oct 30 14:59:40.699554 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:40.699524    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 14:59:40.701085 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:40.701060    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377dfec6_1315_4dd7_81c3_32fbfdc9e6b9.slice/crio-56b6a314286b18c5f803054e4e7522957b07a56b64031bb1438bf0e89180486c WatchSource:0}: Error finding container 56b6a314286b18c5f803054e4e7522957b07a56b64031bb1438bf0e89180486c: Status 404 returned error can't find the container with id 56b6a314286b18c5f803054e4e7522957b07a56b64031bb1438bf0e89180486c
Oct 30 14:59:41.335071 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.334944    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:59:41.335960 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.335666    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:59:41.409140 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.409103    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" event={"ID":"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9","Type":"ContainerStarted","Data":"c126022f4203b19adba7dff9fae20c387094666ba717085a0b7feff96906fbcb"}
Oct 30 14:59:41.409337 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.409158    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" event={"ID":"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9","Type":"ContainerStarted","Data":"56b6a314286b18c5f803054e4e7522957b07a56b64031bb1438bf0e89180486c"}
Oct 30 14:59:41.409337 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.409184    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:41.468739 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:41.468687    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" podStartSLOduration=1.165584309 podStartE2EDuration="1.468671808s" totalImagesPullingTime="303.087499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:41.466168705 +0000 UTC m=+730.908059421" watchObservedRunningTime="2026-10-30 14:59:41.468671808 +0000 UTC m=+730.910562525"
Oct 30 14:59:43.344450 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:43.344418    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 14:59:44.150360 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.150310    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 14:59:44.154500 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.154475    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.175361 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.175328    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 14:59:44.189440 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.187857    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-2c8d89-serving-cert\""
Oct 30 14:59:44.232811 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.232712    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.232811 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.232758    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.233028 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.232862    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr28x\" (UniqueName: \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.233076 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.233048    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.334256 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.334202    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr28x\" (UniqueName: \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.334417    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.334481    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.334513    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:44.334650    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-serving-cert: secret "maas-api-e2e-worker-w6-2c8d89-serving-cert" not found
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:44.334728    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls podName:aabf3fae-02b3-494e-86e0-edf307a702a8 nodeName:}" failed. No retries permitted until 2026-10-30 14:59:44.834709191 +0000 UTC m=+734.276599894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls") pod "maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" (UID: "aabf3fae-02b3-494e-86e0-edf307a702a8") : secret "maas-api-e2e-worker-w6-2c8d89-serving-cert" not found
Oct 30 14:59:44.335810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.335414    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.337491 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.337469    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.364332 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.364259    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr28x\" (UniqueName: \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.450357 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.450322    5064 generic.go:356] "Generic (PLEG): container finished" podID="aaa484c1-ef08-491e-a989-41113905658b" containerID="c1da23b380d51178335418ca0679a2bafb26b00ca9df356a11af454b12d8b253" exitCode=0
Oct 30 14:59:44.450510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.450397    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerDied","Data":"c1da23b380d51178335418ca0679a2bafb26b00ca9df356a11af454b12d8b253"}
Oct 30 14:59:44.450855 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.450813    5064 scope.go:122] "RemoveContainer" containerID="c1da23b380d51178335418ca0679a2bafb26b00ca9df356a11af454b12d8b253"
Oct 30 14:59:44.452288 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.452261    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:44.452358 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.452327    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:44.452395 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.452360    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:44.841982 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.841878    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:44.844805 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:44.844770    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:45.073857 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:45.072072    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:45.408471 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:45.408440    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 14:59:45.410127 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:45.410097    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabf3fae_02b3_494e_86e0_edf307a702a8.slice/crio-d230fa00bd5a14be7cfe1593444c715691124b6c1c2c1ca2065fd1c4ea7d1a7b WatchSource:0}: Error finding container d230fa00bd5a14be7cfe1593444c715691124b6c1c2c1ca2065fd1c4ea7d1a7b: Status 404 returned error can't find the container with id d230fa00bd5a14be7cfe1593444c715691124b6c1c2c1ca2065fd1c4ea7d1a7b
Oct 30 14:59:45.458792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:45.458753    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" event={"ID":"aabf3fae-02b3-494e-86e0-edf307a702a8","Type":"ContainerStarted","Data":"d230fa00bd5a14be7cfe1593444c715691124b6c1c2c1ca2065fd1c4ea7d1a7b"}
Oct 30 14:59:45.461255 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:45.461213    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerStarted","Data":"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"}
Oct 30 14:59:46.311092 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:46.311058    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:46.311281 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:46.311261    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:59:46.311338 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:46.311310    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 14:59:46.482327 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:46.482293    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" event={"ID":"aabf3fae-02b3-494e-86e0-edf307a702a8","Type":"ContainerStarted","Data":"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"}
Oct 30 14:59:46.482825 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:46.482413    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:47.311119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.311085    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:59:47.311308 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.311145    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 14:59:47.378617 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.378588    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 14:59:47.378810 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.378658    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 14:59:47.435854 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.434870    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" podStartSLOduration=3.085622421 podStartE2EDuration="3.434851313s" totalImagesPullingTime="349.228892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:46.541349218 +0000 UTC m=+735.983239934" watchObservedRunningTime="2026-10-30 14:59:47.434851313 +0000 UTC m=+736.876742032"
Oct 30 14:59:47.448328 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.448298    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 14:59:47.448804 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:47.448784    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 14:59:48.290725 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.290687    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 14:59:48.293849 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.293808    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.311005 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.310972    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:59:48.311144 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.311046    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 14:59:48.352859 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.352777    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 14:59:48.353394 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.352985    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-83ff4e-serving-cert\""
Oct 30 14:59:48.389510 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.388168    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 14:59:48.400119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.394606    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.400119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.394764    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqjt\" (UniqueName: \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.400119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.394812    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.400119 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.394874    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.495543 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.495512    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.495791 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.495768    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqjt\" (UniqueName: \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.495902 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.495813    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.495902 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.495892    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.496593 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.496570    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.498621 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.498597    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.499001 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.498985    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.541750 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.541656    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqjt\" (UniqueName: \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\") pod \"maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.609766 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.609737    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:48.957518 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:48.957482    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 14:59:48.958917 ip-10-0-138-143 kubenswrapper[5064]: W1030 14:59:48.958887    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5d297a_1d44_4c2e_ba81_7578644270e7.slice/crio-e2f76635f9241be4293e6663785325b22c30a7eb01bb99dc7b45631e074505fc WatchSource:0}: Error finding container e2f76635f9241be4293e6663785325b22c30a7eb01bb99dc7b45631e074505fc: Status 404 returned error can't find the container with id e2f76635f9241be4293e6663785325b22c30a7eb01bb99dc7b45631e074505fc
Oct 30 14:59:49.310792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.310708    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:59:49.310792 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.310763    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 14:59:49.505483 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.505445    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" event={"ID":"bb5d297a-1d44-4c2e-ba81-7578644270e7","Type":"ContainerStarted","Data":"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"}
Oct 30 14:59:49.505648 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.505500    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" event={"ID":"bb5d297a-1d44-4c2e-ba81-7578644270e7","Type":"ContainerStarted","Data":"e2f76635f9241be4293e6663785325b22c30a7eb01bb99dc7b45631e074505fc"}
Oct 30 14:59:49.505853 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.505770    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:49.509949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.509050    5064 generic.go:356] "Generic (PLEG): container finished" podID="9905b36f-8be0-460c-abc4-acd7404396ba" containerID="fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58" exitCode=0
Oct 30 14:59:49.509949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.509176    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerDied","Data":"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"}
Oct 30 14:59:49.509949 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.509555    5064 scope.go:122] "RemoveContainer" containerID="fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"
Oct 30 14:59:49.511943 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.511881    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:49.512060 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.511959    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:49.512060 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.512009    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:59:49.554814 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:49.554763    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" podStartSLOduration=1.168503353 podStartE2EDuration="1.55475078s" totalImagesPullingTime="386.247427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:49.553555471 +0000 UTC m=+738.995446206" watchObservedRunningTime="2026-10-30 14:59:49.55475078 +0000 UTC m=+738.996641497"
Oct 30 14:59:50.312038 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:50.311293    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:59:50.312038 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:50.311360    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 14:59:50.533885 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:50.528905    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerStarted","Data":"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"}
Oct 30 14:59:51.493177 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.493114    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:51.493751 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.493721    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:59:51.493822 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.493797    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 14:59:51.860903 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.860699    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:51.866853 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.862144    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:51.866853 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:51.863560    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 14:59:52.491476 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:52.491430    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:59:52.491693 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:52.491497    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 14:59:52.497640 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:52.497610    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 14:59:53.491709 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:53.491667    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:59:53.491934 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:53.491726    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 14:59:54.492359 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:54.492322    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:59:54.492750 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:54.492378    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 14:59:55.289858 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:55.288982    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:55.491875 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:55.491840    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:59:55.492029 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:55.491891    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 14:59:55.565110 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:55.565017    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 14:59:56.496278 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:56.496245    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:56.497319 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:56.497292    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 14:59:57.337327 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:57.336963    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:59:57.337327 ip-10-0-138-143 kubenswrapper[5064]: I1030 14:59:57.337035    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:59:59.658730 ip-10-0-138-143 kubenswrapper[5064]: E1030 14:59:59.658689    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:00:00.655399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.655365    5064 generic.go:356] "Generic (PLEG): container finished" podID="28085bb2-5d6a-43fe-b036-af6921999478" containerID="fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59" exitCode=0
Oct 30 15:00:00.655670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.655439    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerDied","Data":"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"}
Oct 30 15:00:00.655896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.655873    5064 scope.go:122] "RemoveContainer" containerID="fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"
Oct 30 15:00:00.657639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.657599    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:00.657717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.657701    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:00.657774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:00.657758    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:01.671551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:01.671518    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerStarted","Data":"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"}
Oct 30 15:00:01.830303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:01.830267    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:00:01.837853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:01.834991    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:01.837853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:01.835067    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:02.831292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:02.831248    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:02.831928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:02.831896    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:03.825714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:03.825263    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:03.825714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:03.825331    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:04.834337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:04.834299    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:04.834980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:04.834945    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:05.699397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.699310    5064 generic.go:356] "Generic (PLEG): container finished" podID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerID="e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e" exitCode=0
Oct 30 15:00:05.699623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.699418    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerDied","Data":"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"}
Oct 30 15:00:05.699934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.699914    5064 scope.go:122] "RemoveContainer" containerID="e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"
Oct 30 15:00:05.701395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.701361    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:05.701484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.701435    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:05.701484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.701464    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:05.830907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.829261    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:05.830907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:05.829323    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:06.707226 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:06.707192    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerStarted","Data":"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"}
Oct 30 15:00:06.835440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:06.835403    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 15:00:06.835731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:06.835703    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 15:00:07.453038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.453003    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:00:07.454550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.454526    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 15:00:07.454618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.454577    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 15:00:07.842534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.842446    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:00:07.843206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.843169    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:00:07.844186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:07.844158    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:00:08.452870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:08.452817    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 15:00:08.453056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:08.452893    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 15:00:09.453207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:09.453170    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 15:00:09.453639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:09.453231    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 15:00:10.453756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:10.453711    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 15:00:10.454211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:10.453782    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 15:00:11.294610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:11.294574    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:00:11.453911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:11.453870    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 15:00:11.454346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:11.453933    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 15:00:12.458403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:12.458372    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:00:12.459433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:12.459409    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:00:29.674945 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:00:29.674881    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:00:38.582362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.582321    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:00:38.586310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.586280    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.605208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.605179    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-357948-openshift-default-dockercfg-b8jx8\""
Oct 30 15:00:38.621689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.621654    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:00:38.644341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644308    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644382    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644430    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644486    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644543    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644602    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg499\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644655    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644681    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.644771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.644761    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746069    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746115    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746143    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746162    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746205    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.746396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746251    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg499\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746459    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746507    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746565    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746672    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746732    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.746942    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.747034    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.747284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.747065    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.749206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.749182    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.749450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.749431    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.771769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.771739    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.771988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.771965    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg499\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:38.899509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:38.899471    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:39.221121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.221091    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:00:39.222139 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:00:39.222100    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e5cde7_d31e_4efb_a40e_5f91ffa0e54e.slice/crio-88777d59fe241f7bd7a0df560754b089326ec2407d6e5767466406dfc4c08c69 WatchSource:0}: Error finding container 88777d59fe241f7bd7a0df560754b089326ec2407d6e5767466406dfc4c08c69: Status 404 returned error can't find the container with id 88777d59fe241f7bd7a0df560754b089326ec2407d6e5767466406dfc4c08c69
Oct 30 15:00:39.224895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.224859    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:39.224986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.224936    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:39.224986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.224968    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:00:39.940687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.940641    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" event={"ID":"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e","Type":"ContainerStarted","Data":"6896a515ba4a5363a830dfe758bcd47ca6a8acc10f6666ea34e6cfcccba16bd5"}
Oct 30 15:00:39.941227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:39.940710    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" event={"ID":"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e","Type":"ContainerStarted","Data":"88777d59fe241f7bd7a0df560754b089326ec2407d6e5767466406dfc4c08c69"}
Oct 30 15:00:40.004948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:40.004887    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podStartSLOduration=2.004871505 podStartE2EDuration="2.004871505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:40.003056387 +0000 UTC m=+789.444947105" watchObservedRunningTime="2026-10-30 15:00:40.004871505 +0000 UTC m=+789.446762222"
Oct 30 15:00:40.900255 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:40.900221    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:40.901917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:40.901892    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 15:00:40.902039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:40.901952    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 15:00:41.900400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:41.900366    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 15:00:41.900790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:41.900415    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 15:00:42.900969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:42.900934    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 15:00:42.901383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:42.900988    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 15:00:43.900865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:43.900809    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 15:00:43.901053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:43.900887    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 15:00:44.900288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:44.900255    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 15:00:44.900473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:44.900305    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 15:00:46.082742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:46.082711    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:46.083283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:46.083212    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:46.084031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:00:46.084013    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:00:59.630109 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:00:59.630071    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:01:06.305319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.305280    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:01:06.311038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.311016    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.331007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.330976    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:01:06.340775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.340737    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"openshift-service-ca.crt\""
Oct 30 15:01:06.340940 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.340819    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"default-dockercfg-6nljl\""
Oct 30 15:01:06.341729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.341702    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"premium-w1-83ff4e-kserve-self-signed-certs\""
Oct 30 15:01:06.341913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.341729    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"kube-root-ca.crt\""
Oct 30 15:01:06.429701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.429673    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.429893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.429712    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.429893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.429742    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.429994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.429897    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.430035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.429995    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.430089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.430046    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6gt\" (UniqueName: \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.538738    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.538811    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.538868    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.538941    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.538997    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6gt\" (UniqueName: \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.539098    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.540253    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.540854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.540490    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.541415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.541070    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.562445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.560471    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.562445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.560752    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.567757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.567725    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6gt\" (UniqueName: \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\") pod \"premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") " pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.639366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.639330    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:06.942395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:06.942366    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:01:06.944223 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:01:06.944199    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8002e35d_46ef_4c8f_a3fa_1adbaee4bf18.slice/crio-338bea5f6cfc8f94e315fdf564e9e2ac6129bea3cee4099fdbdf7bef897c02d8 WatchSource:0}: Error finding container 338bea5f6cfc8f94e315fdf564e9e2ac6129bea3cee4099fdbdf7bef897c02d8: Status 404 returned error can't find the container with id 338bea5f6cfc8f94e315fdf564e9e2ac6129bea3cee4099fdbdf7bef897c02d8
Oct 30 15:01:07.120615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:07.120571    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" event={"ID":"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18","Type":"ContainerStarted","Data":"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"}
Oct 30 15:01:07.120855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:07.120629    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" event={"ID":"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18","Type":"ContainerStarted","Data":"338bea5f6cfc8f94e315fdf564e9e2ac6129bea3cee4099fdbdf7bef897c02d8"}
Oct 30 15:01:07.166735 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:07.166684    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podStartSLOduration=1.166670068 podStartE2EDuration="1.166670068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:07.166133865 +0000 UTC m=+816.608024594" watchObservedRunningTime="2026-10-30 15:01:07.166670068 +0000 UTC m=+816.608560785"
Oct 30 15:01:07.639705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:07.639658    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:09.051092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.051060    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:01:09.058079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.058040    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069501    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069598    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069634    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069667    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069793 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069778    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069840 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069806    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069843    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.069958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069943    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.070010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.069995    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tbh\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.078335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.078309    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-05d396-openshift-default-dockercfg-rlrzr\""
Oct 30 15:01:09.096939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.096887    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:01:09.171069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171028    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171098    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171148    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171205    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171235    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171258    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171314    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171395    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tbh\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171435    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171579    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171637    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171690    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.171935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.171881    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.172216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.172194    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.173426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.173408    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.173661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.173645    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.196541 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.196510    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.196710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.196692    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tbh\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.375311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.375275    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:09.695171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.695141    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:01:09.696032 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:01:09.696003    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ea1616_83a5_4876_bfee_915cf56206ba.slice/crio-d939212beb414cbf0ef668b1f907d8271c61104838ab5815aa452220ff5dd291 WatchSource:0}: Error finding container d939212beb414cbf0ef668b1f907d8271c61104838ab5815aa452220ff5dd291: Status 404 returned error can't find the container with id d939212beb414cbf0ef668b1f907d8271c61104838ab5815aa452220ff5dd291
Oct 30 15:01:09.698461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.698424    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:09.698556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.698507    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:09.698556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:09.698535    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:10.146553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.146517    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" event={"ID":"90ea1616-83a5-4876-bfee-915cf56206ba","Type":"ContainerStarted","Data":"e23f8f1d050138e050b9021024c260d5c7f7609ca6247662d596554e78cf0dc1"}
Oct 30 15:01:10.147029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.146574    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" event={"ID":"90ea1616-83a5-4876-bfee-915cf56206ba","Type":"ContainerStarted","Data":"d939212beb414cbf0ef668b1f907d8271c61104838ab5815aa452220ff5dd291"}
Oct 30 15:01:10.210629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.210563    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podStartSLOduration=1.210542693 podStartE2EDuration="1.210542693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:10.210025247 +0000 UTC m=+819.651915977" watchObservedRunningTime="2026-10-30 15:01:10.210542693 +0000 UTC m=+819.652433414"
Oct 30 15:01:10.376459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.376425    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:10.378036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.378005    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:10.378163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:10.378058    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:11.375748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:11.375712    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:11.376162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:11.375770    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:12.375958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:12.375906    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:12.376467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:12.375984    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:13.376628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:13.376595    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:13.377050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:13.376650    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:14.376728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:14.376685    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:14.377186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:14.376745    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:15.376193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:15.376150    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 15:01:15.376387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:15.376216    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 15:01:16.539862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:16.539817    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:16.540438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:16.540415    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:16.541325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:16.541292    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:01:16.640295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:16.640258    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:16.653335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:16.653309    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:17.222610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:17.222580    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:01:23.408408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.408369    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:01:23.414250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.414221    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.435971 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.435934    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:01:23.487031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.486981    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.487459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487424    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487482    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487535    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487581    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487615    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487665    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q947h\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487776    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.488402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.487842    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589419    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589468    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589508    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589543    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589565    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589601    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589641    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589674    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589713    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q947h\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.589942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589923    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.590185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.589981    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.590400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.590378    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.590482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.590446    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.590595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.590573    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.592218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.592196    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.592536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.592513    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.614684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.614653    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.614843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.614797    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q947h\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.726072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.725966    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:01:23.732185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.732157    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.738405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.738371    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:23.752917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.752882    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:01:23.792562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792525    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792577    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792629    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xh8\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792657    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792672    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792705    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.792765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792744    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.793117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.792973    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.793201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.793175    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894033    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894079    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894167    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894202    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894279    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xh8\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894329    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894353    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894387    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894432    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.894910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.894851    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.895110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.895091    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.895482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.895454    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.895702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.895681    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.895940 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.895922    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.897062 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.897039    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.897686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.897666    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.921488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.921457    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:23.921645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:23.921530    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xh8\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:24.056697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.056610    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:24.079997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.079968    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:01:24.082929 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:01:24.082893    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc861cffd_0939_49dd_8ea8_9f703bb9c942.slice/crio-915e148d91e8e3cd76f7fb5befd7f98186d7f1ec12539ac5495ad6d041f46cac WatchSource:0}: Error finding container 915e148d91e8e3cd76f7fb5befd7f98186d7f1ec12539ac5495ad6d041f46cac: Status 404 returned error can't find the container with id 915e148d91e8e3cd76f7fb5befd7f98186d7f1ec12539ac5495ad6d041f46cac
Oct 30 15:01:24.085804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.085769    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.085907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.085887    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.085948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.085922    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.268342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.268301    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" event={"ID":"c861cffd-0939-49dd-8ea8-9f703bb9c942","Type":"ContainerStarted","Data":"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"}
Oct 30 15:01:24.268613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.268374    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" event={"ID":"c861cffd-0939-49dd-8ea8-9f703bb9c942","Type":"ContainerStarted","Data":"915e148d91e8e3cd76f7fb5befd7f98186d7f1ec12539ac5495ad6d041f46cac"}
Oct 30 15:01:24.356206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.356089    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podStartSLOduration=1.356070924 podStartE2EDuration="1.356070924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:24.353878001 +0000 UTC m=+833.795768718" watchObservedRunningTime="2026-10-30 15:01:24.356070924 +0000 UTC m=+833.797961642"
Oct 30 15:01:24.429670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.429625    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:01:24.431236 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:01:24.431182    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dfeb13_f0d5_4f47_b20d_c3a131443ae9.slice/crio-5cc8c5dff9348c4457686f4df7bd5a4aaefd24d13beb067024ed0c64fb772913 WatchSource:0}: Error finding container 5cc8c5dff9348c4457686f4df7bd5a4aaefd24d13beb067024ed0c64fb772913: Status 404 returned error can't find the container with id 5cc8c5dff9348c4457686f4df7bd5a4aaefd24d13beb067024ed0c64fb772913
Oct 30 15:01:24.434139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.434097    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.434248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.434195    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.434398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.434246    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:24.748231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:24.748200    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:25.278396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:25.278357    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" event={"ID":"03dfeb13-f0d5-4f47-b20d-c3a131443ae9","Type":"ContainerStarted","Data":"f3d6b9f022282a201329619a26c877425e854e49e6ee4fd5ae39795e0640e55a"}
Oct 30 15:01:25.278575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:25.278416    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" event={"ID":"03dfeb13-f0d5-4f47-b20d-c3a131443ae9","Type":"ContainerStarted","Data":"5cc8c5dff9348c4457686f4df7bd5a4aaefd24d13beb067024ed0c64fb772913"}
Oct 30 15:01:25.380100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:25.380034    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podStartSLOduration=2.380015766 podStartE2EDuration="2.380015766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:25.376878144 +0000 UTC m=+834.818768873" watchObservedRunningTime="2026-10-30 15:01:25.380015766 +0000 UTC m=+834.821906483"
Oct 30 15:01:25.742786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:25.742747    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:25.743490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:25.743448    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:26.069908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:26.059212    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:26.071007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:26.070967    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:26.071147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:26.071037    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:26.740629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:26.740595    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:26.757706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:26.757658    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:27.066462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:27.066365    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:27.066749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:27.066711    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:27.762856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:27.761993    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:27.762856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:27.762086    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:28.060344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:28.060243    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:28.060639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:28.060608    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:28.746638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:28.746596    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:28.746947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:28.746917    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:29.060330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:29.060237    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:29.061049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:29.060917    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:29.678438 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:01:29.678401    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:01:29.741856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:29.741257    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:29.741856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:29.741318    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:30.070283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:30.070185    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:30.072701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:30.072659    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:30.739025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:30.738984    5064 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 15:01:30.739234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:30.739057    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 15:01:31.057735 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:31.057641    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 15:01:31.057735 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:31.057711    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 15:01:31.745015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:31.744981    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:31.745540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:31.745265    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:31.746072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:31.746051    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:01:32.344519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:32.344484    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:32.344889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:32.344858    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:32.345771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:32.345751    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:01:54.354008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.353932    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:01:54.358516 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.358494    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.375575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.375546    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4b703c-openshift-default-dockercfg-f85tr\""
Oct 30 15:01:54.379092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.379065    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:01:54.442238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442201    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442311    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442349    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442404    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkq4\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442432    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442682 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442663    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442717    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442844    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.442979 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.442941    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.543912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.543869    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.543919    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.543962    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544025    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkq4\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544051    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544163    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544209    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544252    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544307    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544323    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544372    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544466    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.544731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.544655    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.545308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.545283    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.547223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.547202    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.547548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.547525    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.569651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.569621    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.569806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.569701    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkq4\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:54.705502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:54.705463    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:55.029995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.029968    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:01:55.031475 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:01:55.031443    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032a0871_0784_4069_9be1_8bb563921baf.slice/crio-fa41a7553bf1a8788b956079da8cf4906c0cc52889f075c8bfbd196b3e7f8d3b WatchSource:0}: Error finding container fa41a7553bf1a8788b956079da8cf4906c0cc52889f075c8bfbd196b3e7f8d3b: Status 404 returned error can't find the container with id fa41a7553bf1a8788b956079da8cf4906c0cc52889f075c8bfbd196b3e7f8d3b
Oct 30 15:01:55.033916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.033881    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:55.033997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.033946    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:55.033997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.033981    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:01:55.598853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.598508    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" event={"ID":"032a0871-0784-4069-9be1-8bb563921baf","Type":"ContainerStarted","Data":"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"}
Oct 30 15:01:55.598853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.598600    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" event={"ID":"032a0871-0784-4069-9be1-8bb563921baf","Type":"ContainerStarted","Data":"fa41a7553bf1a8788b956079da8cf4906c0cc52889f075c8bfbd196b3e7f8d3b"}
Oct 30 15:01:55.700808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.700134    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podStartSLOduration=1.70011072 podStartE2EDuration="1.70011072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:55.679905792 +0000 UTC m=+865.121796510" watchObservedRunningTime="2026-10-30 15:01:55.70011072 +0000 UTC m=+865.142001437"
Oct 30 15:01:55.706558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:55.706522    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:01:56.714856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:56.714249    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:01:56.714856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:56.714328    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:01:57.706933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:57.706895    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:01:57.707219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:57.707184    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:01:58.706635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:58.706600    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:01:58.707051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:58.706652    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:01:59.651757 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:01:59.651720    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:01:59.706516 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:59.706483    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:01:59.706690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:01:59.706544    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:02:00.706556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:00.706519    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:02:00.706747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:00.706578    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:02:01.706011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:01.705975    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 15:02:01.706204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:01.706033    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 15:02:03.160666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:03.160630    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:02:03.161140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:03.161054    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:02:03.162056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:03.162033    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:02:08.866090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.866039    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:02:08.872533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.872503    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.902656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.902624    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991272    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6dn\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991418    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991458    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991519    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991553    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991688    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991720    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:08.992993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:08.991787    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.027779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.027740    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:02:09.032038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.032021    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.054551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.054523    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:02:09.092820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.092790    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.092820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.092822    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.092891    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.092945    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6dn\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093014    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093034    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093087    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093118    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093216    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093298    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093494    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093757    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093777    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.093985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.093919    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.095421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.095402    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.095672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.095655    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.118294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.118234    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6dn\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.119273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.119252    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.194477 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194433    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.194687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194586    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.194687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194666    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:09.194918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194705    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.194918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194747    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.194918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194776    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.194918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194868    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xgx\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.195128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.194981    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.195128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.195045    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.195202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.195128    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296702    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296766    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296806    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296856    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296878    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xgx\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.296945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296930    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.296980    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297047    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297091    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297359    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297512    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297540    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297567    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.297893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.297814    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.299676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.299658    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.299854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.299821    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.322226 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.322192    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.323033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.322998    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xgx\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.346856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.346809    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:09.516341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.516309    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:02:09.518221 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:09.518188    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13f4b1c_8da8_4b6e_94cb_78beb8086558.slice/crio-9e85e9a85083ab785f50e65970751eac876ab8a9bfc476f47f9ed1dc777c3011 WatchSource:0}: Error finding container 9e85e9a85083ab785f50e65970751eac876ab8a9bfc476f47f9ed1dc777c3011: Status 404 returned error can't find the container with id 9e85e9a85083ab785f50e65970751eac876ab8a9bfc476f47f9ed1dc777c3011
Oct 30 15:02:09.520721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.520686    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.520799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.520755    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.520799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.520785    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.657641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.657612    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:02:09.661070 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:09.661039    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe822e50_9c5e_4233_893a_20179742182d.slice/crio-df2c3856369c3748130c186147da462f0625fffe97722f68cfaaf6771c40fe1b WatchSource:0}: Error finding container df2c3856369c3748130c186147da462f0625fffe97722f68cfaaf6771c40fe1b: Status 404 returned error can't find the container with id df2c3856369c3748130c186147da462f0625fffe97722f68cfaaf6771c40fe1b
Oct 30 15:02:09.663075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.663037    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.663182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.663126    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.663182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.663173    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:09.774599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.774528    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" event={"ID":"f13f4b1c-8da8-4b6e-94cb-78beb8086558","Type":"ContainerStarted","Data":"a6ac4cbab3e014bc22d6fbe8828ef17a9a3e67657a4f8bada60294bc0f99dedf"}
Oct 30 15:02:09.774599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.774602    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" event={"ID":"f13f4b1c-8da8-4b6e-94cb-78beb8086558","Type":"ContainerStarted","Data":"9e85e9a85083ab785f50e65970751eac876ab8a9bfc476f47f9ed1dc777c3011"}
Oct 30 15:02:09.776966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.776935    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" event={"ID":"fe822e50-9c5e-4233-893a-20179742182d","Type":"ContainerStarted","Data":"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"}
Oct 30 15:02:09.777133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.776989    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" event={"ID":"fe822e50-9c5e-4233-893a-20179742182d","Type":"ContainerStarted","Data":"df2c3856369c3748130c186147da462f0625fffe97722f68cfaaf6771c40fe1b"}
Oct 30 15:02:09.828019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.827962    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podStartSLOduration=1.8279473419999999 podStartE2EDuration="1.827947342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:09.826911642 +0000 UTC m=+879.268802366" watchObservedRunningTime="2026-10-30 15:02:09.827947342 +0000 UTC m=+879.269838058"
Oct 30 15:02:09.876965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:09.876916    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podStartSLOduration=0.876896093 podStartE2EDuration="876.896093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:09.875138913 +0000 UTC m=+879.317029634" watchObservedRunningTime="2026-10-30 15:02:09.876896093 +0000 UTC m=+879.318786809"
Oct 30 15:02:10.195683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.195591    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:10.197314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.197178    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:10.197314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.197228    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:10.347634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.347482    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:10.353311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.352935    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:10.353311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:10.352991    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:11.195553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:11.195513    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:11.195965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:11.195572    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:11.359857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:11.355436    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:11.359857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:11.355499    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:12.195769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:12.195738    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:12.196169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:12.195791    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:12.356855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:12.347256    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:12.356855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:12.347323    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:13.195310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:13.195270    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:13.195495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:13.195332    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:13.354855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:13.353974    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:13.354855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:13.354041    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:14.195596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:14.195556    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:14.195790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:14.195615    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:14.347349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:14.347307    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:14.347651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:14.347617    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:15.195573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:15.195530    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:15.196099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:15.195589    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:15.362855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:15.352978    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:15.362855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:15.353039    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:16.216865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:16.213970    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 15:02:16.216865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:16.214037    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 15:02:16.351857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:16.347888    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 15:02:16.351857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:16.347966    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 15:02:17.374852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:17.374323    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:17.376859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:17.376175    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:17.385855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:17.385015    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:02:18.389996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:18.389951    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:02:18.390495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:18.390019    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:02:18.783589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:18.783496    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:18.783898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:18.783873    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:18.784818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:18.784796    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:02:26.007465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.007409    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:02:26.021324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.021282    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.027701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.027664    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:02:26.039908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.039881    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"openshift-service-ca.crt\""
Oct 30 15:02:26.052479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.052446    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"kube-root-ca.crt\""
Oct 30 15:02:26.052652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.052494    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"default-dockercfg-576h8\""
Oct 30 15:02:26.052652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.052629    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"distinct-2-w3-357948-kserve-self-signed-certs\""
Oct 30 15:02:26.118256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118217    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.118481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118384    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.118565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118481    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.118565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118545    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9sqj\" (UniqueName: \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.118672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118572    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.118751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.118729    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224035    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224277    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224371    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224422    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224456    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9sqj\" (UniqueName: \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.224485    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.226673    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.231857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.226981    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.239849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.234159    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.251234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.251135    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.270984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.269239    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.284211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.272795    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9sqj\" (UniqueName: \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\") pod \"distinct-2-w3-357948-kserve-679d8584d9-kbxbd\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") " pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.345743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.345713    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:26.652937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:26.652908    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:02:26.653712 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:26.653689    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad20a01_4a33_461c_9919_fbcbda8f6158.slice/crio-5f15788b3d0a2e1e8661252d32aac54503fb0f10211425442236560c9c4e133c WatchSource:0}: Error finding container 5f15788b3d0a2e1e8661252d32aac54503fb0f10211425442236560c9c4e133c: Status 404 returned error can't find the container with id 5f15788b3d0a2e1e8661252d32aac54503fb0f10211425442236560c9c4e133c
Oct 30 15:02:27.015974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:27.015878    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" event={"ID":"dad20a01-4a33-461c-9919-fbcbda8f6158","Type":"ContainerStarted","Data":"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"}
Oct 30 15:02:27.015974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:27.015923    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" event={"ID":"dad20a01-4a33-461c-9919-fbcbda8f6158","Type":"ContainerStarted","Data":"5f15788b3d0a2e1e8661252d32aac54503fb0f10211425442236560c9c4e133c"}
Oct 30 15:02:27.060910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:27.060824    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podStartSLOduration=2.060807703 podStartE2EDuration="2.060807703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:27.060086055 +0000 UTC m=+896.501976773" watchObservedRunningTime="2026-10-30 15:02:27.060807703 +0000 UTC m=+896.502698421"
Oct 30 15:02:27.346453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:27.346344    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:29.631254 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:02:29.631203    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:02:30.805583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:30.805554    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:02:30.807996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:30.807967    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:02:30.877759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:30.877722    5064 scope.go:122] "RemoveContainer" containerID="4d37f820f204bc1bdd7e2eb29f1346be3b952d5132596d96f0eae0e128224a5d"
Oct 30 15:02:36.346485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:36.346451    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:36.359686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:36.359660    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:37.172195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:37.172167    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:02:39.344200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.344138    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:02:39.350681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.350656    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.367966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.367935    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:02:39.372364 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.372335    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:02:39.372611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.372488    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.396773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.396735    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:02:39.496625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.496586    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:02:39.500872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.500820    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.511806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.511775    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.511932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.511879    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.511932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.511923    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.511954    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.511983    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512008    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512080    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512125    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512189    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512267    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512319    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512361    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512390    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512414    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512450    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkb8g\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512479    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.512558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512501    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4sv\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.512558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.512533    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.539221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.539173    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:02:39.613443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613356    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkb8g\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613399    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613416    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4sv\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.613749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613544    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.613749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613593    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613655    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.613749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613692    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613818    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613861    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613895    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613922    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613940    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.613990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.613966    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614022    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614122    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614170    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614197    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614229    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614231    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614314    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614323    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614346    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614372    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614398    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614430    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614548    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614600    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614611    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614633    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8mg\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614647    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614704    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614737    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614762    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.614928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614813    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.615293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614931    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.615293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.614957    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.615293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.615155    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.616466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.616442    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.616752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.616732    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.617032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.617015    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.617126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.617110    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.649970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.649916    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkb8g\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.650114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.650070    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4sv\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.652139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.652116    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.652257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.652212    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.665311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.665288    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:39.695216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.695190    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:39.716216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716185    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716317    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716355    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716395    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716423    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716520    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8mg\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716584    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716619    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716654    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716705    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716844    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.716919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.716651    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.717130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.717110    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.717332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.717314    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.719053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.719032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.719140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.719128    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.747328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.747294    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.750269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.749725    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8mg\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.818310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.817457    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:39.994337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.994304    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:02:39.995305 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:39.995279    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf46a028_001f_4536_b3c3_b324b54634cd.slice/crio-ef55c5bba3208ff36e02933c3178a903cb603ae5a04aa326c019b197e5b760cf WatchSource:0}: Error finding container ef55c5bba3208ff36e02933c3178a903cb603ae5a04aa326c019b197e5b760cf: Status 404 returned error can't find the container with id ef55c5bba3208ff36e02933c3178a903cb603ae5a04aa326c019b197e5b760cf
Oct 30 15:02:39.997826 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.997776    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:39.997974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.997877    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:39.997974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:39.997925    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.022603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.022579    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:02:40.032057 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:40.032036    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b453573_d86d_43c2_81d0_b7c74aee2706.slice/crio-a8083f74c0da38b103f3b3a1afdced24c4c3655f81e135afe6d208c9c4582551 WatchSource:0}: Error finding container a8083f74c0da38b103f3b3a1afdced24c4c3655f81e135afe6d208c9c4582551: Status 404 returned error can't find the container with id a8083f74c0da38b103f3b3a1afdced24c4c3655f81e135afe6d208c9c4582551
Oct 30 15:02:40.034523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.034493    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.034613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.034561    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.034613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.034593    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.139302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.139210    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:02:40.149606 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:40.149562    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a14ea05_11b3_4e04_ae5e_e9e01648c8c8.slice/crio-7481f96333b496945072ed784b26a30c5ffab20b35467d9b4e0db5f303bdfeb2 WatchSource:0}: Error finding container 7481f96333b496945072ed784b26a30c5ffab20b35467d9b4e0db5f303bdfeb2: Status 404 returned error can't find the container with id 7481f96333b496945072ed784b26a30c5ffab20b35467d9b4e0db5f303bdfeb2
Oct 30 15:02:40.153642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.153592    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.153794 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.153692    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.153794 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.153740    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:40.184746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.184233    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" event={"ID":"df46a028-001f-4536-b3c3-b324b54634cd","Type":"ContainerStarted","Data":"5ef376a19a4bad988f70afa4bb48303b17771ffe73eeb964f6c25bf5a1de8d03"}
Oct 30 15:02:40.184746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.184293    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" event={"ID":"df46a028-001f-4536-b3c3-b324b54634cd","Type":"ContainerStarted","Data":"ef55c5bba3208ff36e02933c3178a903cb603ae5a04aa326c019b197e5b760cf"}
Oct 30 15:02:40.187020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.186992    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" event={"ID":"5b453573-d86d-43c2-81d0-b7c74aee2706","Type":"ContainerStarted","Data":"32f8a46c427807d14cdf1dad9fb7f7b1f732ab67a15fc0c5d77fb0a1062acf82"}
Oct 30 15:02:40.187140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.187043    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" event={"ID":"5b453573-d86d-43c2-81d0-b7c74aee2706","Type":"ContainerStarted","Data":"a8083f74c0da38b103f3b3a1afdced24c4c3655f81e135afe6d208c9c4582551"}
Oct 30 15:02:40.188729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.188705    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" event={"ID":"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8","Type":"ContainerStarted","Data":"7481f96333b496945072ed784b26a30c5ffab20b35467d9b4e0db5f303bdfeb2"}
Oct 30 15:02:40.236211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.236165    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podStartSLOduration=1.236150836 podStartE2EDuration="1.236150836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:40.233820374 +0000 UTC m=+909.675711106" watchObservedRunningTime="2026-10-30 15:02:40.236150836 +0000 UTC m=+909.678041554"
Oct 30 15:02:40.287852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.286803    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podStartSLOduration=1.286783952 podStartE2EDuration="1.286783952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:40.284484479 +0000 UTC m=+909.726375199" watchObservedRunningTime="2026-10-30 15:02:40.286783952 +0000 UTC m=+909.728674670"
Oct 30 15:02:40.666188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.666146    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:40.696014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:40.695985    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:41.200465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.199898    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" event={"ID":"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8","Type":"ContainerStarted","Data":"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"}
Oct 30 15:02:41.268851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.265034    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podStartSLOduration=2.265015278 podStartE2EDuration="2.265015278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:41.264502708 +0000 UTC m=+910.706393425" watchObservedRunningTime="2026-10-30 15:02:41.265015278 +0000 UTC m=+910.706905996"
Oct 30 15:02:41.687865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.686980    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:41.687865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.687048    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:41.697787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.697756    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:41.697977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.697806    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:41.831850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.825880    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:41.831850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.830943    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:41.831850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:41.831001    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:42.676854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.676503    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:42.676854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.676566    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:42.695946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.695910    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:42.696337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.695974    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:42.848450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.848405    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:42.848732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:42.848700    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:43.672869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.672332    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:43.672869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.672406    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:43.696512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.696472    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:43.696949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.696528    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:43.829855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.821301    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:43.829855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:43.821372    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:44.671856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.668276    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:44.671856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.668345    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:44.696263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.696224    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:44.696422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.696292    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:44.819285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.819251    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:44.819902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:44.819873    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:45.666673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.666630    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:45.667017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.666986    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:45.696427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.696386    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:45.696601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.696461    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:45.880851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.822006    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:45.907981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:45.885132    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:46.665807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.665762    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 15:02:46.666057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.665845    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 15:02:46.696112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.696069    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 15:02:46.696285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.696141    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 15:02:46.818645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.818609    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 15:02:46.818850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:46.818664    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 15:02:47.683670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.683631    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:47.684473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.684451    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:47.687559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.687536    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:02:47.700083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.700062    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:47.700357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.700329    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:47.701050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.701030    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:02:47.824264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.824227    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:47.825286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.825230    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:47.825966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:47.825940    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:02:54.008948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.008908    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:02:54.013340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.013316    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.053006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.052939    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:02:54.107276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107207    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107258    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107412    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107512    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcj7\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107580    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107671    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107775    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107894    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.107945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.107922    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.212949    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213008    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213034    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213157    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213182    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213255    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213307    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcj7\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213361    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.213852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.213398    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.247850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.234309    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.247850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.235008    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.247850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.235774    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.247850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.236454    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.247850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.236973    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.291439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.286584    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.292802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.292769    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcj7\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.297966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.297401    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.297966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.297912    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.335426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.335391    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:54.686595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.685919    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:02:54.686595 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:54.685946    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425ae39f_9069_43e9_98f1_671f1eaafc5b.slice/crio-86c03675f6fcc6e0d1ed1f7da3b5a05ab07b72819ccb4ac51184cb8ab43abf9c WatchSource:0}: Error finding container 86c03675f6fcc6e0d1ed1f7da3b5a05ab07b72819ccb4ac51184cb8ab43abf9c: Status 404 returned error can't find the container with id 86c03675f6fcc6e0d1ed1f7da3b5a05ab07b72819ccb4ac51184cb8ab43abf9c
Oct 30 15:02:54.688994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.688949    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:54.689094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.689043    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:54.689139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:54.689093    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:02:55.498286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:55.498234    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" event={"ID":"425ae39f-9069-43e9-98f1-671f1eaafc5b","Type":"ContainerStarted","Data":"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"}
Oct 30 15:02:55.498775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:55.498343    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" event={"ID":"425ae39f-9069-43e9-98f1-671f1eaafc5b","Type":"ContainerStarted","Data":"86c03675f6fcc6e0d1ed1f7da3b5a05ab07b72819ccb4ac51184cb8ab43abf9c"}
Oct 30 15:02:55.587662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:55.587589    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podStartSLOduration=2.587568058 podStartE2EDuration="2.587568058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:55.569291831 +0000 UTC m=+925.011182562" watchObservedRunningTime="2026-10-30 15:02:55.587568058 +0000 UTC m=+925.029458775"
Oct 30 15:02:56.335628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:56.335552    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:02:56.340643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:56.340602    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:02:56.340846 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:56.340686    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:02:57.180615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.180575    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:02:57.186343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.186312    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.207376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.207338    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:02:57.233855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.217173    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"distinct-2-w1-83ff4e-kserve-self-signed-certs\""
Oct 30 15:02:57.338856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.338270    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:02:57.338856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.338330    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:02:57.344657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.344621    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lm2\" (UniqueName: \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.344813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.344764    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.344930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.344914    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.344990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.344939    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.345086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.345069    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.345122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.345110    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.453855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.453734    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.454140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.454110    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.454369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.454351    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.454503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.454490    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.454660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.454639    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lm2\" (UniqueName: \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.454866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.454847    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.455502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.455478    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.455923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.455903    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.477984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.475432    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.481864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.480324    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.491050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.486558    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.558674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.558599    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lm2\" (UniqueName: \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\") pod \"distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:57.805900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:57.805804    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:58.142299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.142264    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:02:58.143444 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:02:58.143406    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183f88c0_d9d3_4195_a7a2_a0ed402f2265.slice/crio-348b90de2f75cb3171d702fcc175ec15f2656cfd409ffad506e19e40a4fea527 WatchSource:0}: Error finding container 348b90de2f75cb3171d702fcc175ec15f2656cfd409ffad506e19e40a4fea527: Status 404 returned error can't find the container with id 348b90de2f75cb3171d702fcc175ec15f2656cfd409ffad506e19e40a4fea527
Oct 30 15:02:58.338322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.338283    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:02:58.338770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.338347    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:02:58.549912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.549799    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" event={"ID":"183f88c0-d9d3-4195-a7a2-a0ed402f2265","Type":"ContainerStarted","Data":"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"}
Oct 30 15:02:58.549912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.549873    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" event={"ID":"183f88c0-d9d3-4195-a7a2-a0ed402f2265","Type":"ContainerStarted","Data":"348b90de2f75cb3171d702fcc175ec15f2656cfd409ffad506e19e40a4fea527"}
Oct 30 15:02:58.597693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.597623    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podStartSLOduration=1.597606437 podStartE2EDuration="1.597606437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:58.594817843 +0000 UTC m=+928.036708557" watchObservedRunningTime="2026-10-30 15:02:58.597606437 +0000 UTC m=+928.039497154"
Oct 30 15:02:58.806668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:58.806587    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:02:59.336855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:59.336232    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:02:59.336855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:02:59.336295    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:02:59.651370 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:02:59.651330    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:03:00.338267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:00.338229    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:03:00.338465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:00.338283    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:03:01.336163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:01.336124    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 15:03:01.336648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:01.336189    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 15:03:02.697052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:02.697006    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:03:02.698414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:02.698390    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:03:02.700166 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:02.700143    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:03:07.806312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:07.806267    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:07.819554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:07.819517    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:08.637507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:08.637475    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:09.195814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.195781    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:03:09.200417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.200392    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.224631 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.224594    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:03:09.270000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.269957    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7jx\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270155    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270338    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270473    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270560    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270663    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270694    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270792    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.270915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.270896    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372327    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372411    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372514    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372558    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372641    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372667    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372723    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372770    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.372962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372847    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7jx\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.373072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.372970    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.373446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.373005    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.373628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.373561    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.373744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.373718    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.373811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.373743    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.376243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.376218    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.376655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.376628    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.399093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.399056    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.399238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.399119    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7jx\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.525618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.525526    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:09.850101 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.850075    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:03:09.850646 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:03:09.850618    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942d2e88_2f2e_48b9_bf7c_11b0402667e2.slice/crio-8c7080e52ab1f8bfe92a10bd7695dfc9851436b8bd525f6edd9f5141d6d80c18 WatchSource:0}: Error finding container 8c7080e52ab1f8bfe92a10bd7695dfc9851436b8bd525f6edd9f5141d6d80c18: Status 404 returned error can't find the container with id 8c7080e52ab1f8bfe92a10bd7695dfc9851436b8bd525f6edd9f5141d6d80c18
Oct 30 15:03:09.853326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.853290    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:09.853428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.853365    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:09.853428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:09.853420    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:10.645965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:10.645930    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" event={"ID":"942d2e88-2f2e-48b9-bf7c-11b0402667e2","Type":"ContainerStarted","Data":"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"}
Oct 30 15:03:10.646367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:10.645981    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" event={"ID":"942d2e88-2f2e-48b9-bf7c-11b0402667e2","Type":"ContainerStarted","Data":"8c7080e52ab1f8bfe92a10bd7695dfc9851436b8bd525f6edd9f5141d6d80c18"}
Oct 30 15:03:10.697949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:10.697898    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podStartSLOduration=1.697884699 podStartE2EDuration="1.697884699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:10.696445083 +0000 UTC m=+940.138335811" watchObservedRunningTime="2026-10-30 15:03:10.697884699 +0000 UTC m=+940.139775416"
Oct 30 15:03:11.533476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:11.533436    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:11.538010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:11.537970    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:11.538212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:11.538037    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:12.535861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:12.535238    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:12.535861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:12.535319    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:13.533277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:13.533237    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:13.533571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:13.533539    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:14.534854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:14.534315    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:14.534854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:14.534393    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:15.537992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:15.537939    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:15.538466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:15.538013    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:16.530332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:16.530293    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:16.530600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:16.530574    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:17.526854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:17.525879    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 15:03:17.526854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:17.525955    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 15:03:18.234555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:18.234503    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:03:18.751060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:18.751004    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:03:18.868400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:18.854372    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:03:18.868400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:18.854454    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:19.353961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:19.353925    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:19.354773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:19.354672    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:19.361058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:19.361021    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:03:29.632327 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:03:29.632284    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:03:37.191856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:37.191054    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:37.230061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:37.230014    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:37.350976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:37.350922    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:37.743761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:37.723951    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:03:37.772124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:37.745755    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:38.302856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.279969    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:38.333855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.305792    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:03:38.479780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.332544    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:03:38.483170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.483142    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:03:38.683114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.677763    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:03:38.690719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.690694    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:03:38.803854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.782714    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:38.861637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.861584    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:38.894728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.888934    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:38.894728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:38.891488    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:39.967378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.967341    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:03:39.975539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.975512    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.985926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.985890    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.985986    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986004    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986053    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986168    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzcx\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986217    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986256    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986286    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986308    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:39.986624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.986572    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:03:39.991665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.991640    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:39.999716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:39.999685    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:03:40.025575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.025539    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:03:40.087235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087201    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087316    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087352    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087374    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.087419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087401    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.087634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087606    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087619    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.087737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087662    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzcx\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087709    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.087879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087746    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.087956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087882    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.087956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087928    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.088075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.087970    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088025    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.088075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088067    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088103    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088193    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.088234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088223    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088235    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088317    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5x9h\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.088587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088400    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088530    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.088729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.088709    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.090265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.090246    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.090416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.090378    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.115562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.115530    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.115714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.115620    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzcx\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.189427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189385    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5x9h\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189491    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189519    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189639    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189674    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189722    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189788    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.189943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189894    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.189967    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.190023    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.190059    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.190210    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.190246    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.190606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.190580    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.192094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.192070    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.192277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.192258    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.215481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.215451    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.215595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.215578    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5x9h\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.297804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.297713    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:40.308415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.308389    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:40.642789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.642758    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:03:40.644365 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:03:40.644338    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be869ab_5708_4180_80f7_801d1a44ec28.slice/crio-ddd91e8a38f0e8c7a1b6f311b518db69bfc0e0ef39d1884e82d49a3f64c69417 WatchSource:0}: Error finding container ddd91e8a38f0e8c7a1b6f311b518db69bfc0e0ef39d1884e82d49a3f64c69417: Status 404 returned error can't find the container with id ddd91e8a38f0e8c7a1b6f311b518db69bfc0e0ef39d1884e82d49a3f64c69417
Oct 30 15:03:40.646322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.646297    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:03:40.646955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.646923    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:40.647052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.647000    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:40.647052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.647043    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:40.659422 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:03:40.659394    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15983fe3_07d4_440d_9447_b1122fab9c45.slice/crio-6a60e8bf71ebb53e0da0926ff21b936ef265da342287ada013e4ed8c1b472965 WatchSource:0}: Error finding container 6a60e8bf71ebb53e0da0926ff21b936ef265da342287ada013e4ed8c1b472965: Status 404 returned error can't find the container with id 6a60e8bf71ebb53e0da0926ff21b936ef265da342287ada013e4ed8c1b472965
Oct 30 15:03:40.661958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.661929    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:40.662036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.661992    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:40.662036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:40.662025    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:41.569396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.569356    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" event={"ID":"3be869ab-5708-4180-80f7-801d1a44ec28","Type":"ContainerStarted","Data":"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"}
Oct 30 15:03:41.569882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.569414    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" event={"ID":"3be869ab-5708-4180-80f7-801d1a44ec28","Type":"ContainerStarted","Data":"ddd91e8a38f0e8c7a1b6f311b518db69bfc0e0ef39d1884e82d49a3f64c69417"}
Oct 30 15:03:41.571042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.571013    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" event={"ID":"15983fe3-07d4-440d-9447-b1122fab9c45","Type":"ContainerStarted","Data":"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"}
Oct 30 15:03:41.571143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.571057    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" event={"ID":"15983fe3-07d4-440d-9447-b1122fab9c45","Type":"ContainerStarted","Data":"6a60e8bf71ebb53e0da0926ff21b936ef265da342287ada013e4ed8c1b472965"}
Oct 30 15:03:41.661904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.655606    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podStartSLOduration=2.655584054 podStartE2EDuration="2.655584054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:41.637331635 +0000 UTC m=+971.079222355" watchObservedRunningTime="2026-10-30 15:03:41.655584054 +0000 UTC m=+971.097474772"
Oct 30 15:03:41.700713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:41.700661    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podStartSLOduration=2.7006465779999997 podStartE2EDuration="2.700646578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:41.697198714 +0000 UTC m=+971.139089431" watchObservedRunningTime="2026-10-30 15:03:41.700646578 +0000 UTC m=+971.142537294"
Oct 30 15:03:42.298452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.298420    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:42.299672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.299646    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:42.299739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.299704    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:42.309065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.308962    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:42.310314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.310281    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:42.310408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:42.310350    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:43.344902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:43.324226    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:43.344902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:43.324310    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:43.344902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:43.324981    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:43.344902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:43.325021    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:44.298886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:44.298852    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:44.299059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:44.298935    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:44.309129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:44.309099    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:44.309252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:44.309148    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:45.298641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:45.298601    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:45.299053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:45.298659    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:45.309448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:45.309415    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:45.309596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:45.309464    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:46.298332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:46.298294    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:46.298542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:46.298358    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:46.308789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:46.308762    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:46.309164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:46.308810    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:47.298559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:47.298512    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 15:03:47.298775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:47.298593    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 15:03:47.309067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:47.309032    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 15:03:47.309416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:47.309087    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 15:03:48.401079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.401045    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:48.402635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.402611    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:48.403920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.403894    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:03:48.439577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.439544    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:48.440912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.440881    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:48.445536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:48.443238    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:03:51.689854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:51.687542    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:52.114191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:52.113694    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:52.846855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:52.838009    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:03:53.114648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:53.114546    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:53.605576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:53.605542    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:03:53.683926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:53.667983    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:53.750198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:53.735352    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:53.895854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:53.892010    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:54.038254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.011044    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:03:54.038534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.038499    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:54.152911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.118107    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:03:54.314614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.305002    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:03:54.503762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.503557    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:03:54.508455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.508413    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.517636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.517602    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:03:54.517898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.517879    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:03:54.518038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.517979    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.599147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.598401    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:03:54.676853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.651144    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.681490    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.681771    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.681924    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2pjs\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682096    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682253    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682369    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682532    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682698    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.682727    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683001    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683139    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683171    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g69g\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683339    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683500    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683659    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683800    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.693895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.683825    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.807269    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809469    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809505    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809579    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809626    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809652    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g69g\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809710    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809763    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809809    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809871    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.809895    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810050    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810117    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810176    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810244    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2pjs\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810335    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810393    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810418    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.810895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.810517    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.818352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.818319    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.819129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.819105    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.819859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.819819    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.821320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.821301    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:03:54.821492 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.821481    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:03:54.822285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.822263    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.825783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.822602    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.825969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.824081    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.826093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.824636    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.826205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.824790    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.826308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.825059    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.826413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.825284    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.826511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.826246    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.827077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.827060    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.831507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.830454    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.832083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.831899    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:03:54.833099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.833051    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.875086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.872942    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:54.876402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.876371    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2pjs\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.877368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.877342    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:54.878532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:54.878506    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g69g\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:55.055490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.055394    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:55.130559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.130528    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:55.400909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.400861    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:03:55.405090 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:03:55.405054    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fa1e63_f074_4f35_a83f_80896ce4dea2.slice/crio-9fb78d33fff02436a912303e73c6f095481737bdb081fefa463ea81cc7ded915 WatchSource:0}: Error finding container 9fb78d33fff02436a912303e73c6f095481737bdb081fefa463ea81cc7ded915: Status 404 returned error can't find the container with id 9fb78d33fff02436a912303e73c6f095481737bdb081fefa463ea81cc7ded915
Oct 30 15:03:55.408169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.408127    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.408375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.408228    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.408375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.408273    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.461037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.461005    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:03:55.461845 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:03:55.461797    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41932f6a_cb4d_4856_8d4b_39b49adb3bba.slice/crio-47341813992d1d1f616527a7bd6666083afbc8981ff9f702c99c114da0452579 WatchSource:0}: Error finding container 47341813992d1d1f616527a7bd6666083afbc8981ff9f702c99c114da0452579: Status 404 returned error can't find the container with id 47341813992d1d1f616527a7bd6666083afbc8981ff9f702c99c114da0452579
Oct 30 15:03:55.464413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.464378    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.464509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.464448    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.464509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.464476    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:03:55.524673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.524639    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" event={"ID":"41932f6a-cb4d-4856-8d4b-39b49adb3bba","Type":"ContainerStarted","Data":"47341813992d1d1f616527a7bd6666083afbc8981ff9f702c99c114da0452579"}
Oct 30 15:03:55.526648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.526613    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" event={"ID":"80fa1e63-f074-4f35-a83f-80896ce4dea2","Type":"ContainerStarted","Data":"a3513121822819c9e4c74a1d6519bdf87553b554f1ce38cf819384aa589ce543"}
Oct 30 15:03:55.526816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.526674    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" event={"ID":"80fa1e63-f074-4f35-a83f-80896ce4dea2","Type":"ContainerStarted","Data":"9fb78d33fff02436a912303e73c6f095481737bdb081fefa463ea81cc7ded915"}
Oct 30 15:03:55.578311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:55.578255    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podStartSLOduration=2.578237047 podStartE2EDuration="2.578237047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:55.575316157 +0000 UTC m=+985.017206876" watchObservedRunningTime="2026-10-30 15:03:55.578237047 +0000 UTC m=+985.020127765"
Oct 30 15:03:56.060321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:56.060228    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:03:56.063924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:56.063463    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:03:56.063924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:56.063561    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:03:56.579249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:56.579197    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" event={"ID":"41932f6a-cb4d-4856-8d4b-39b49adb3bba","Type":"ContainerStarted","Data":"e3034fad677d9e921467512bfdd580457c402944b2f0dfb6e47daa76cf59232b"}
Oct 30 15:03:56.676886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:56.672614    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podStartSLOduration=2.6725916830000003 podStartE2EDuration="2.672591683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:56.654345341 +0000 UTC m=+986.096236081" watchObservedRunningTime="2026-10-30 15:03:56.672591683 +0000 UTC m=+986.114482401"
Oct 30 15:03:57.061985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:57.061944    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:03:57.062585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:57.062550    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:03:57.138929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:57.138889    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:03:57.161857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:57.143882    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:03:57.162923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:57.162879    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:03:58.056855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:58.056173    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:03:58.056855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:58.056238    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:03:58.135865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:58.133149    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:03:58.135865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:58.133213    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:03:59.060855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:59.057175    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:03:59.060855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:59.057237    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:03:59.143858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:59.142976    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:03:59.143858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:03:59.143050    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:03:59.674739 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:03:59.674696    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:04:00.059916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:00.058264    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:04:00.059916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:00.058336    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:04:00.134305 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:00.134264    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:04:00.134620 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:00.134587    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:04:01.060855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:01.060226    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:04:01.060855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:01.060307    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:04:01.132855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:01.132300    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:04:01.132855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:01.132396    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:04:02.056342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:02.056304    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 15:04:02.056595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:02.056364    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 15:04:02.131199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:02.131163    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 15:04:02.131599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:02.131230    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 15:04:03.337790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.337754    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:04:03.339155 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.339133    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:04:03.340174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.340154    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:04:03.422427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.422395    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:04:03.423147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.423111    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:04:03.424352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:03.424330    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:04:09.181210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.181166    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:04:09.196068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.196034    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:04:09.196403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.196384    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.324932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.324882    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.325249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.325206    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.325537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.325502    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2mg\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.325738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.325720    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.325937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.325903    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.326166 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.326149    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.338876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.331877    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.338876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.331941    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.338876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.332049    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.443565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.443464    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.443898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.443871    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.444087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.444070    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.444308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.444289    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.444507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.444489    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.444682 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.444655    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.444958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.444938    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2mg\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.445162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.445130    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.445324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.445304    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.484898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.467129    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.487110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.487075    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.487591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.487566    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.488278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.488256    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.502866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.497334    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.532163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.532129    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.541858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.541538    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2mg\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.557575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.554474    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.625625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.625548    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:09.995596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:09.984556    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:04:10.106009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:10.103939    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:10.106009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:10.104536    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:10.253097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:10.251009    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:11.150014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.149963    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:04:11.152290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.152253    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:11.338349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.338293    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:11.641434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.641401    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:04:11.641736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.641716    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:04:11.641917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.641900    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:04:11.664672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.664641    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:04:11.940458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:11.940076    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:04:12.182373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.182262    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:04:12.424032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.422279    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:04:12.430536 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:04:12.430490    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e0daa3_f81e_4b68_9c4c_ff683d9d1de8.slice/crio-68a5d02ed392d6f7c16bf9bd19fec5ae4feb2e27dcde47c1155fa69bd3ec5b37 WatchSource:0}: Error finding container 68a5d02ed392d6f7c16bf9bd19fec5ae4feb2e27dcde47c1155fa69bd3ec5b37: Status 404 returned error can't find the container with id 68a5d02ed392d6f7c16bf9bd19fec5ae4feb2e27dcde47c1155fa69bd3ec5b37
Oct 30 15:04:12.432905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.432871    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:04:12.433007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.432963    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:04:12.433068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.433033    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:04:12.664660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.664616    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" event={"ID":"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8","Type":"ContainerStarted","Data":"4b7ae868088a871c8cb235f6d243a338da96701778eaa1da3eeed2a75ff2e6a7"}
Oct 30 15:04:12.666487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.666457    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" event={"ID":"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8","Type":"ContainerStarted","Data":"68a5d02ed392d6f7c16bf9bd19fec5ae4feb2e27dcde47c1155fa69bd3ec5b37"}
Oct 30 15:04:12.715022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:12.714914    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podStartSLOduration=3.714896465 podStartE2EDuration="3.714896465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:12.714339146 +0000 UTC m=+1002.156229867" watchObservedRunningTime="2026-10-30 15:04:12.714896465 +0000 UTC m=+1002.156787184"
Oct 30 15:04:13.147401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:13.147364    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:13.149042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:13.149015    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:13.149186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:13.149067    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:14.148206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:14.148173    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:14.148618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:14.148231    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:15.147554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:15.147517    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:15.147789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:15.147597    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:16.147415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:16.147376    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:16.147872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:16.147435    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:17.180845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:17.179984    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:17.180845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:17.180047    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:18.147632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:18.147593    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:18.147853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:18.147653    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:19.147819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:19.147781    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 15:04:19.148447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:19.148411    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 15:04:20.187163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:20.187125    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:20.189106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:20.189078    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:20.189346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:20.189329    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:04:29.633041 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:04:29.633002    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:04:33.720855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:33.719953    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:34.106401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.106290    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:34.207988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.207932    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:04:34.420343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.419941    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:04:34.420343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.420013    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:34.483999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.483956    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:04:34.484306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.484270    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:34.723956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.722919    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:34.849094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:34.831299    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:04:35.114918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.113981    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:35.135866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.115575    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:04:35.218856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.218266    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:04:35.220186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.219189    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:04:35.496852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.493959    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:04:35.570695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.570663    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:04:35.757068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:35.744547    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:04:56.751217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:56.751114    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.351982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.351917    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.616915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.613974    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.643854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.632799    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.698077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.698021    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.715272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.715053    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:04:57.715272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.715144    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:57.842480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.801074    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:04:57.842480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:57.801525    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:04:58.641077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.634497    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:58.641077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.637791    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:04:58.663858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.661041    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:04:58.686054 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.677631    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:04:58.859012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.858945    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 15:04:58.902193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:58.868779    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:04:59.638416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:59.636849    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:04:59.663446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:04:59.663094    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:04:59.874856 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:04:59.854515    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.05s"
Oct 30 15:05:00.112047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:00.111934    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:05:00.296857 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:00.280002    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:05:00.648319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:00.648276    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:00.716854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:00.674018    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:05:01.489561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:01.113445    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:05:01.819914 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:01.792032    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:01.920851 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:01.920076    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.203s"
Oct 30 15:05:02.011858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:02.011372    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:02.371115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:02.371089    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:02.379533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:02.379369    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:17.659860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.656856    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9022-a9aa55"]
Oct 30 15:05:17.693590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.693545    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:17.724418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.724378    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:17.725344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.725322    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\""
Oct 30 15:05:17.767915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.767883    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-a9aa55"]
Oct 30 15:05:17.848888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.835391    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2hq\" (UniqueName: \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\") pod \"test-curl-9022-a9aa55\" (UID: \"955cde51-819e-4642-80e2-b2594d3927c8\") " pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:17.953595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.953497    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2hq\" (UniqueName: \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\") pod \"test-curl-9022-a9aa55\" (UID: \"955cde51-819e-4642-80e2-b2594d3927c8\") " pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:18.003856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:17.993441    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2hq\" (UniqueName: \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\") pod \"test-curl-9022-a9aa55\" (UID: \"955cde51-819e-4642-80e2-b2594d3927c8\") " pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:18.159317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.158065    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:18.323863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.307928    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:05:18.423856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.422950    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:18.423856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.423028    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:18.447856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.446927    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:18.447856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.446993    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:18.690885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.681127    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:18.690885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:18.681977    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:19.175857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.157710    5064 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:19.250858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.237026    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:19.564857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.557711    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 15:05:19.686875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.686797    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:19.688768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.688602    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:05:19.695666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:19.689891    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:05:20.105872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.080394    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:20.426857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.412420    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.489531    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:20.448366    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.731s"
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.489640    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-a9aa55"]
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.489850    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.489910    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:20.497306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.489972    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:20.537195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.537136    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:20.586624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.564722    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:20.944856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:20.930120    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:05:21.000609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.000560    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:05:21.095473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.095401    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:21.316502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.316411    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:21.316758 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.316728    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:21.742703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.731606    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-a9aa55" event={"ID":"955cde51-819e-4642-80e2-b2594d3927c8","Type":"ContainerStarted","Data":"629da4956020a203b2f8a132e5135c14c5a38dc8e7e427349f4c6db65c915b64"}
Oct 30 15:05:21.943856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.925800    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx" podUID="034782e3-3b6e-4c3b-852b-ea4b23f06a75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:21.967865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:21.945395    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:05:22.037170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:22.024902    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:22.118902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:22.097102    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:22.988854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:22.985476    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:23.075920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:23.025722    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:23.128780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:23.128754    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:23.129493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:23.129045    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:23.149697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:23.149671    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:25.719282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:25.719240    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-a9aa55" event={"ID":"955cde51-819e-4642-80e2-b2594d3927c8","Type":"ContainerStarted","Data":"30c981c1180985e68926e2fb521c77e5fa9620cbad8080e5b1073162023c3fac"}
Oct 30 15:05:25.764515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:25.764453    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9022-a9aa55" podStartSLOduration=3.715650403 podStartE2EDuration="8.764438234s" totalImagesPullingTime="5.048787831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:25.761716737 +0000 UTC m=+1075.203607454" watchObservedRunningTime="2026-10-30 15:05:25.764438234 +0000 UTC m=+1075.206328950"
Oct 30 15:05:26.676879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:26.676824    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9022-a9aa55"]
Oct 30 15:05:26.681379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:26.681348    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9022-a9aa55"]
Oct 30 15:05:27.732935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:27.732888    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9022-a9aa55" podUID="955cde51-819e-4642-80e2-b2594d3927c8" containerName="test-curl-9022-a9aa55" containerID="cri-o://30c981c1180985e68926e2fb521c77e5fa9620cbad8080e5b1073162023c3fac" gracePeriod=2
Oct 30 15:05:29.632204 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:29.632157    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:05:29.808962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:29.808908    5064 generic.go:356] "Generic (PLEG): container finished" podID="955cde51-819e-4642-80e2-b2594d3927c8" containerID="30c981c1180985e68926e2fb521c77e5fa9620cbad8080e5b1073162023c3fac" exitCode=137
Oct 30 15:05:29.991837 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:29.991796    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:30.084375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.084343    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\" (UniqueName: \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\") pod \"955cde51-819e-4642-80e2-b2594d3927c8\" (UID: \"955cde51-819e-4642-80e2-b2594d3927c8\") "
Oct 30 15:05:30.086531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.086488    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq" pod "955cde51-819e-4642-80e2-b2594d3927c8" (UID: "955cde51-819e-4642-80e2-b2594d3927c8"). InnerVolumeSpecName "kube-api-access-wp2hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:05:30.186713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.186677    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2hq\" (UniqueName: \"kubernetes.io/projected/955cde51-819e-4642-80e2-b2594d3927c8-kube-api-access-wp2hq\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:05:30.725137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.725105    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955cde51-819e-4642-80e2-b2594d3927c8" path="/var/lib/kubelet/pods/955cde51-819e-4642-80e2-b2594d3927c8/volumes"
Oct 30 15:05:30.818985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.818955    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-a9aa55"
Oct 30 15:05:30.819175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:30.819003    5064 scope.go:122] "RemoveContainer" containerID="30c981c1180985e68926e2fb521c77e5fa9620cbad8080e5b1073162023c3fac"
Oct 30 15:05:33.020976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.020917    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:33.028368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.028326    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:33.139858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.111505    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:05:33.160853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.141308    5064 cpu_manager.go:414] "containerMap: removing container" podUID="955cde51-819e-4642-80e2-b2594d3927c8" containerName="test-curl-9022-a9aa55"
Oct 30 15:05:33.162380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.162352    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955cde51-819e-4642-80e2-b2594d3927c8" containerName="test-curl-9022-a9aa55"
Oct 30 15:05:33.168885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.133246    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:33.178887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.175912    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="955cde51-819e-4642-80e2-b2594d3927c8" containerName="test-curl-9022-a9aa55"
Oct 30 15:05:33.239549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.239512    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:05:33.251951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.245882    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.273365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.273179    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c4cd83d6-gw-openshift-default-dockercfg-npv9b\""
Oct 30 15:05:33.495915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.495865    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.496301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.496272    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.496549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.496529    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.496800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.496780    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.497009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.496990    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.497234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.497215    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.497477 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.497458    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.497659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.497642    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.509060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.509000    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhr5\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614198    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614277    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614318    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614387    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614417    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614470    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614556    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614588    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.615004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.614648    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhr5\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.641367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.641300    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.641620 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.641594    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.642324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.642293    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.642755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.642727    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.675563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.669916    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.676200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.676117    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.685857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.682794    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.774864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.766499    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.774864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.770715    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhr5\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\") pod \"e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") " pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.813949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.796173    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:33.894474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.894424    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:33.894728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:33.894681    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:34.032902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.032078    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:05:34.033430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.032990    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:34.036858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.035471    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:34.036858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.035564    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:34.074855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.074762    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:34.245336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.243020    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:34.245336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.243121    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:34.393019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.392942    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:34.393311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.393255    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:34.449517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.448394    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:34.499917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.473995    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:34.499917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:34.474063    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:35.377933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:35.353985    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:35.535856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:35.532023    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:35.606747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:35.606674    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:36.172640 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:36.172506    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.381s"
Oct 30 15:05:36.276232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.276188    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerStarted","Data":"f68d7e6692bc5447ca37a0b7606e1fd9654ff49821d1da159a6e9f13437fb97e"}
Oct 30 15:05:36.299892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.297401    5064 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:36.299892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.297468    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:36.301714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.301672    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:36.347177 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.315664    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:36.387856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.369964    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:05:36.580105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:36.580004    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:37.426582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:37.426526    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:37.710127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:37.647888    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 15:05:37.710127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:37.648226    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:05:37.716766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:37.716721    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 15:05:37.747499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:37.747455    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:05:38.407019 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:38.397643    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.682s"
Oct 30 15:05:38.407019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.397952    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:38.407019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.398111    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:38.655059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.654595    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:38.723973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.723514    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:38.782953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.725724    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:05:38.783409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.760494    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:38.866125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.866074    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 15:05:38.915304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.915210    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:38.915570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:38.915529    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:39.742416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:39.742315    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:39.839607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:39.839547    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:40.095922 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:40.083546    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.301s"
Oct 30 15:05:40.096370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.096330    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:40.132626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.132503    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:05:40.284881 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.237436    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:05:40.328050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.328008    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerStarted","Data":"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"}
Oct 30 15:05:40.343611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.343575    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:05:40.345430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.248079    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.352538    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podStartSLOduration=7.352516671 podStartE2EDuration="7.352516671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:40.328543644 +0000 UTC m=+1089.770434395" watchObservedRunningTime="2026-10-30 15:05:40.352516671 +0000 UTC m=+1089.794407390"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357415    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357465    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357500    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357558    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357612    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8kl\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357647    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357800    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357886    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.357970    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.363900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.363373    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:40.363900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.363407    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:05:40.364184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.364161    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.365638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.365609    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:40.372920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.370708    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3834be83-openshift-default-dockercfg-ntlpx\""
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533540    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533600    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533634    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533673    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533713    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8kl\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533747    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.533919    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.534011    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.534075    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.559299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.551415    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.607754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.567689    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.607754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.588952    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dc34df04-openshift-default-dockercfg-8sx6h\""
Oct 30 15:05:40.607754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.593849    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.619947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.618618    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.619947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.619001    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.629573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.629540    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.639167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.639135    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.657848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.657793    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.658256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.658229    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.659897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.659870    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8kl\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.663578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.663533    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.663891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.663814    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.664259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.664239    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.664731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.664701    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.665055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.665034    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.665147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.665102    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.665227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.665206    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5db6\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.667207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.667185    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\") pod \"e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") " pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:40.766405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766370    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.766860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766518    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.766860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766596    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.766860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766625    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.766860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766687    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5db6\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.766860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766757    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766880    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766904    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766947    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.766975    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.767084    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.767200    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.767342    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.767549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.767441    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.769497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.769474    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.769696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.769676    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.792487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.792451    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.792640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.792490    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5db6\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\") pod \"e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") " pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.797268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.797242    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:40.802927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.802900    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:40.833657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.833619    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:40.934263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:40.933890    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:41.202406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.202287    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:05:41.203270 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:05:41.203231    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ee1a63_d18b_4de0_b303_a73e06fba4a4.slice/crio-b08503d525498ba3126537a2e2e4adb084cfa2591f454189b0f75f839794a45b WatchSource:0}: Error finding container b08503d525498ba3126537a2e2e4adb084cfa2591f454189b0f75f839794a45b: Status 404 returned error can't find the container with id b08503d525498ba3126537a2e2e4adb084cfa2591f454189b0f75f839794a45b
Oct 30 15:05:41.206474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.206436    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:41.206587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.206536    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:41.206659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.206586    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:41.212517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.212489    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerStarted","Data":"b08503d525498ba3126537a2e2e4adb084cfa2591f454189b0f75f839794a45b"}
Oct 30 15:05:41.212796 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.212773    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:41.213916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.213897    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:05:41.256946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.256911    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:05:41.257057 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:05:41.256940    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ac1230_a258_4779_ba13_6e23970da19a.slice/crio-f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56 WatchSource:0}: Error finding container f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56: Status 404 returned error can't find the container with id f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56
Oct 30 15:05:41.260580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.260544    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:41.260671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.260617    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:41.260671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:41.260650    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:42.221863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.221813    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerStarted","Data":"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"}
Oct 30 15:05:42.222380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.221896    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerStarted","Data":"f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56"}
Oct 30 15:05:42.223790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.223765    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerStarted","Data":"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"}
Oct 30 15:05:42.302946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.302892    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podStartSLOduration=4.302878756 podStartE2EDuration="4.302878756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:42.285741517 +0000 UTC m=+1091.727632234" watchObservedRunningTime="2026-10-30 15:05:42.302878756 +0000 UTC m=+1091.744769467"
Oct 30 15:05:42.335115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.335061    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podStartSLOduration=3.335043986 podStartE2EDuration="3.335043986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:42.333906751 +0000 UTC m=+1091.775797471" watchObservedRunningTime="2026-10-30 15:05:42.335043986 +0000 UTC m=+1091.776934703"
Oct 30 15:05:42.834304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.834268    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:42.840577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.840543    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:42.934497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.934462    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:42.940706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:42.940678    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:43.231502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:43.231474    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:43.232024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:43.231874    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:43.232798 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:43.232780    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:05:43.232898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:43.232847    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:05:53.514855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:53.513752    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:05:53.855911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:53.854939    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:53.899854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:53.898972    5064 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:05:53.899854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:53.899044    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:54.530504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:54.530455    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:54.533411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:54.533383    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:54.928318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:54.913618    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:54.952859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:54.928973    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:55.374267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:55.367007    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:55.561856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:55.561084    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:56.355024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.354972    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:56.374784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.374752    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:05:56.391894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.389541    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:56.613103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.612983    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:56.847991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.847960    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:56.875394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:56.875353    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9qz5r" podUID="f009e6e3-5135-48c0-864c-c9672de9b17a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.239560    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9r8\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.239716    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.239752    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.239800    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.240073    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.240173    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.240221    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.240310    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.240911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.240391    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.460623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.460575    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:57.461870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.459823    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx" podUID="034782e3-3b6e-4c3b-852b-ea4b23f06a75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.578790    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.578941    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.578988    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579075    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579198    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579263    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9r8\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579376    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579430    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.580962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.579469    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.659854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.631279    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.659854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.631497    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 15:05:57.659854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.631870    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:05:57.659854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.632775    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 15:05:57.661007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.660907    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.661474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.661421    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.802885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.798193    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.949550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.949505    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.970265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.969186    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:57.979864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:57.974708    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 15:05:58.005518 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.003284    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:58.009350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.009317    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:58.038946 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:58.034966    5064 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.213s"
Oct 30 15:05:58.038946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.035044    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:58.038946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.035072    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:05:58.038946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.035114    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:05:58.049419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.048496    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9r8\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:58.061480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.061433    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:05:58.061480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.061484    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:05:58.061728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.061554    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.081721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.081677    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:05:58.098099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.096457    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:05:58.111052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.111027    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:05:58.245045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.244206    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.245045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.244325    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.245045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.244547    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.245045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.244591    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.361978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.361941    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.362225 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.362204    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.362396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.362381    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.362560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.362548    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhf4\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.362743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.362723    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464127    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464166    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464230    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhf4\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464281    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464332    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464386    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464463    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.464722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.464486    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.466168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.465298    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.466168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.465497    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.466732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.466563    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.466732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.466653    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.466939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.466813    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.468057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.468032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.468705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.468684    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.494265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.494235    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.499214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.499145    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhf4\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:58.663747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.663714    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:05:58.664194 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:05:58.664168    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874909f2_47cb_4dee_ba41_0744403bc5e9.slice/crio-62d2242a5d2c6527a315cbaa837275b47312befe0257a5e17273b42915ac278f WatchSource:0}: Error finding container 62d2242a5d2c6527a315cbaa837275b47312befe0257a5e17273b42915ac278f: Status 404 returned error can't find the container with id 62d2242a5d2c6527a315cbaa837275b47312befe0257a5e17273b42915ac278f
Oct 30 15:05:58.666868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.666824    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:58.666933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.666902    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:58.666933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.666929    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:58.734620 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:58.734596    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:05:59.063752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.063723    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:05:59.064526 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:05:59.064500    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda508efa7_be16_48ac_9a84_d12f05a6ac29.slice/crio-42e109202a8829362e46a0065e87ccee49485adb986f1adf1cafaf2bb93d26d3 WatchSource:0}: Error finding container 42e109202a8829362e46a0065e87ccee49485adb986f1adf1cafaf2bb93d26d3: Status 404 returned error can't find the container with id 42e109202a8829362e46a0065e87ccee49485adb986f1adf1cafaf2bb93d26d3
Oct 30 15:05:59.067993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.067954    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:59.068110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.068025    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:59.068110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.068056    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:05:59.457121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.457081    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" event={"ID":"a508efa7-be16-48ac-9a84-d12f05a6ac29","Type":"ContainerStarted","Data":"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"}
Oct 30 15:05:59.457306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.457141    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" event={"ID":"a508efa7-be16-48ac-9a84-d12f05a6ac29","Type":"ContainerStarted","Data":"42e109202a8829362e46a0065e87ccee49485adb986f1adf1cafaf2bb93d26d3"}
Oct 30 15:05:59.459037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.459008    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" event={"ID":"874909f2-47cb-4dee-ba41-0744403bc5e9","Type":"ContainerStarted","Data":"cd2caf0c32ac91c975630e04e0b59e6ac9c93bba3365d81676d4a534ee9af2a6"}
Oct 30 15:05:59.459037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.459052    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" event={"ID":"874909f2-47cb-4dee-ba41-0744403bc5e9","Type":"ContainerStarted","Data":"62d2242a5d2c6527a315cbaa837275b47312befe0257a5e17273b42915ac278f"}
Oct 30 15:05:59.508880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.508804    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podStartSLOduration=4.50878989 podStartE2EDuration="4.50878989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:59.505806022 +0000 UTC m=+1108.947696730" watchObservedRunningTime="2026-10-30 15:05:59.50878989 +0000 UTC m=+1108.950680607"
Oct 30 15:05:59.565360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.565300    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podStartSLOduration=5.565284345 podStartE2EDuration="5.565284345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:59.560781482 +0000 UTC m=+1109.002672200" watchObservedRunningTime="2026-10-30 15:05:59.565284345 +0000 UTC m=+1109.007175068"
Oct 30 15:05:59.651222 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:05:59.651183    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:05:59.738383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:05:59.738027    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:06:00.113907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:00.112938    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:06:00.122855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:00.121657    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:00.122855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:00.121737    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:00.770513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:00.770468    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:00.771009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:00.770795    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:01.112900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:01.112764    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:01.112900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:01.112825    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:01.758500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:01.758460    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:01.758801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:01.758773    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:02.112135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.112048    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:02.112135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.112104    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:02.161866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.159148    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:02.183926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.183885    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.187037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.187007    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:02.201287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.201258    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c4cd83d6-serving-cert\""
Oct 30 15:06:02.301135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.301099    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.301544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.301515    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.301871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.301850    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.302109 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.302055    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jqm\" (UniqueName: \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.564255    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.564487    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.564568    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jqm\" (UniqueName: \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.564633    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:02.573015    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-serving-cert: secret "maas-api-e2e-migrate-c4cd83d6-serving-cert" not found
Oct 30 15:06:02.579005 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:02.573107    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls podName:b25f5dea-9b81-4cea-a2bf-587c65120a9c nodeName:}" failed. No retries permitted until 2026-10-30 15:06:03.073080991 +0000 UTC m=+1112.514971703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls") pod "maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" (UID: "b25f5dea-9b81-4cea-a2bf-587c65120a9c") : secret "maas-api-e2e-migrate-c4cd83d6-serving-cert" not found
Oct 30 15:06:02.581615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.581581    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.604343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.603604    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.660658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.660624    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jqm\" (UniqueName: \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:02.749308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.749265    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:02.749560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:02.749531    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:03.103202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.103163    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:03.107576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.107547    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") pod \"maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:03.135018 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.134969    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:03.135528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.135043    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:03.179986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.179956    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:03.670905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.670640    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:03.754268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.754229    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:03.770997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:03.770953    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:04.120865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.112865    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:04.120865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.112953    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:04.751855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.751524    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:04.751855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.751583    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:04.801790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.801746    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" event={"ID":"b25f5dea-9b81-4cea-a2bf-587c65120a9c","Type":"ContainerStarted","Data":"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"}
Oct 30 15:06:04.802005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.801815    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:04.802005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.801858    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" event={"ID":"b25f5dea-9b81-4cea-a2bf-587c65120a9c","Type":"ContainerStarted","Data":"d779d979f7a58e3e53073bdd03a30af4d8b2eecd80cc076a4c875da452786102"}
Oct 30 15:06:04.812559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:04.812513    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" podStartSLOduration=2.428751204 podStartE2EDuration="2.812500564s" totalImagesPullingTime="383.74936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:04.809096981 +0000 UTC m=+1114.250987701" watchObservedRunningTime="2026-10-30 15:06:04.812500564 +0000 UTC m=+1114.254391334"
Oct 30 15:06:05.185898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:05.159741    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 15:06:05.187575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:05.187537    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 15:06:05.735746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:05.735703    5064 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 15:06:05.735992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:05.735789    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 15:06:06.548379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.548335    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:06:06.551079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.551022    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:06:06.552188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.552149    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:06:06.784318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.784283    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:06:06.785199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.785173    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:06:06.787010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:06.786990    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:06:08.344517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.344478    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:08.349354 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.349332    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.368452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.368422    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:08.385872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.385823    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3834be83-serving-cert\""
Oct 30 15:06:08.465932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.465890    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.466303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.466277    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.466441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.466419    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97bs\" (UniqueName: \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.466517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.466497    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.590278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.590236    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.601138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.601039    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97bs\" (UniqueName: \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.609993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.609945    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.610374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.610347    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.619523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.619484    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.631414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.631322    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.669331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.669294    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97bs\" (UniqueName: \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.672471 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.672433    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\") pod \"maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:08.966182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:08.966138    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:09.288906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:09.280756    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:09.882200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:09.880168    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" event={"ID":"f368bc6a-b67a-405e-a572-a88ff29deea3","Type":"ContainerStarted","Data":"fdc57fed16b783e025974de969ef57bc006a895558e105e64184b6ed1e1faecc"}
Oct 30 15:06:10.771324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:10.771240    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:10.893051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:10.893015    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" event={"ID":"f368bc6a-b67a-405e-a572-a88ff29deea3","Type":"ContainerStarted","Data":"0bdaa1279f78fc3a6c84b8aa25b2aad7c1da4a3d9cedc6da6256620cd2bfc4f1"}
Oct 30 15:06:10.893454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:10.893104    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:10.954194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:10.954143    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" podStartSLOduration=2.397111529 podStartE2EDuration="2.954126713s" totalImagesPullingTime="557.015184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:10.950856704 +0000 UTC m=+1120.392747422" watchObservedRunningTime="2026-10-30 15:06:10.954126713 +0000 UTC m=+1120.396017430"
Oct 30 15:06:16.907079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:16.907049    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:22.393857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:22.357019    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:22.614857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:22.590573    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:06:23.095857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:23.091933    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:23.662021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:23.661967    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:23.663345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:23.663314    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:06:24.136926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:24.120143    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:24.136926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:24.124154    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:06:24.509986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:24.506964    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:24.673004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:24.672957    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 15:06:24.903292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:24.895468    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 15:06:25.135852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:25.132970    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:06:25.306808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:25.306710    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:06:25.577915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:25.535804    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:06:25.634859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:25.634506    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy" containerID="cri-o://f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2" gracePeriod=30
Oct 30 15:06:29.274060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:29.274015    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:06:29.274567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:29.274106    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:29.274567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:29.274436    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:06:29.274567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:29.274475    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:29.638196 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:29.638119    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:06:31.034687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.034586    5064 scope.go:122] "RemoveContainer" containerID="c1da23b380d51178335418ca0679a2bafb26b00ca9df356a11af454b12d8b253"
Oct 30 15:06:31.156194 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:31.156119    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:31.584266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.584184    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 15:06:31.664723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.664678    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.665742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.665693    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666248    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666412    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666476    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666506    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666542    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666597    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.666817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.666798    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\") pod \"aaa484c1-ef08-491e-a989-41113905658b\" (UID: \"aaa484c1-ef08-491e-a989-41113905658b\") "
Oct 30 15:06:31.668260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.668139    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:31.668362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.668296    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:31.668587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.668552    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:31.668683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.668584    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:06:31.669499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.669468    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:31.672430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.671603    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "kube-api-access-mj4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:31.677585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.677538    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:31.679082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.679052    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:31.682284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.682234    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo" pod "aaa484c1-ef08-491e-a989-41113905658b" (UID: "aaa484c1-ef08-491e-a989-41113905658b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.780895    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa484c1-ef08-491e-a989-41113905658b-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.780984    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781001    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa484c1-ef08-491e-a989-41113905658b-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781014    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781027    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781049    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781064    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4w4\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-kube-api-access-mj4w4\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781076    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa484c1-ef08-491e-a989-41113905658b-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.781117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.781090    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa484c1-ef08-491e-a989-41113905658b-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:31.799891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.799857    5064 generic.go:356] "Generic (PLEG): container finished" podID="aaa484c1-ef08-491e-a989-41113905658b" containerID="f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2" exitCode=0
Oct 30 15:06:31.800110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.799901    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"
Oct 30 15:06:31.800110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.799904    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerDied","Data":"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"}
Oct 30 15:06:31.800110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.799960    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj" event={"ID":"aaa484c1-ef08-491e-a989-41113905658b","Type":"ContainerDied","Data":"35b87760b6a1cac67e73dc9e3adca1a9347fa26e506ef79736e062f0cc00663d"}
Oct 30 15:06:31.800110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.799984    5064 scope.go:122] "RemoveContainer" containerID="f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"
Oct 30 15:06:31.812105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.812075    5064 scope.go:122] "RemoveContainer" containerID="f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"
Oct 30 15:06:31.812489 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:31.812464    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2\": container with ID starting with f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2 not found: ID does not exist" containerID="f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"
Oct 30 15:06:31.812571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.812501    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2"} err="failed to get container status \"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2\": rpc error: code = NotFound desc = could not find container \"f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2\": container with ID starting with f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2 not found: ID does not exist"
Oct 30 15:06:31.890547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.890506    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 15:06:31.905744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:31.905712    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-2wznj"]
Oct 30 15:06:32.762030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:32.761942    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa484c1-ef08-491e-a989-41113905658b" path="/var/lib/kubelet/pods/aaa484c1-ef08-491e-a989-41113905658b/volumes"
Oct 30 15:06:33.818691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.818657    5064 generic.go:356] "Generic (PLEG): container finished" podID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerID="16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051" exitCode=0
Oct 30 15:06:33.819163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.818722    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerDied","Data":"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"}
Oct 30 15:06:33.819260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.819241    5064 scope.go:122] "RemoveContainer" containerID="16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"
Oct 30 15:06:33.820621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.820600    5064 generic.go:356] "Generic (PLEG): container finished" podID="62ac1230-a258-4779-ba13-6e23970da19a" containerID="75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823" exitCode=0
Oct 30 15:06:33.820703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.820650    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerDied","Data":"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"}
Oct 30 15:06:33.821105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.821072    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.821199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.821146    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.821199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.821185    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.821312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.821293    5064 scope.go:122] "RemoveContainer" containerID="75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"
Oct 30 15:06:33.822941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.822913    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.823029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.822979    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.823029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.823014    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.823246 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.823224    5064 generic.go:356] "Generic (PLEG): container finished" podID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerID="161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157" exitCode=0
Oct 30 15:06:33.823304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.823290    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerDied","Data":"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"}
Oct 30 15:06:33.823664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.823646    5064 scope.go:122] "RemoveContainer" containerID="161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"
Oct 30 15:06:33.825184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.825152    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.825259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.825236    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:33.825305 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:33.825284    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:35.004872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.004816    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerStarted","Data":"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"}
Oct 30 15:06:35.036867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.030339    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerStarted","Data":"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"}
Oct 30 15:06:35.061862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.049304    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerStarted","Data":"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"}
Oct 30 15:06:35.594742 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:35.582386    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:35.800261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.800150    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:35.800554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.800527    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:35.800753 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.800726    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:35.867942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.860253    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:06:35.867942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.860388    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:35.867942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.860449    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:35.969854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.960315    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:06:35.969854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.961823    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:35.969854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:35.961899    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:36.802645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.802519    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:36.802645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.802565    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:36.899312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.878648    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:36.899312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.878920    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:36.951163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.944838    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:36.951163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:36.944917    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:37.810486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.810447    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:37.811148 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.811096    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:37.882913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.882083    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:37.882913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.882148    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:37.955259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.955217    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:37.955548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:37.955518    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:38.797582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.797178    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:38.797582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.797231    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:38.844853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.841556    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:38.844853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.841620    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:38.949300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.949257    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:38.949594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:38.949567    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:39.797098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.797057    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:39.797418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.797369    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:39.845594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.845547    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:39.846260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.846216    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:39.954947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.946636    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:39.954947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:39.946712    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:40.797657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.797624    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 15:06:40.797882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.797690    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 15:06:40.851871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.849325    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 15:06:40.851871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.849403    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 15:06:40.934625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.934589    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 15:06:40.935120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:40.934868    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 15:06:41.212661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.212628    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:41.255415 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:41.255376    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:41.801575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.801545    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:41.802786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.802766    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:41.839766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.839732    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:06:41.840041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.840010    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:06:41.840792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.840768    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:06:41.942159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.942125    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:06:41.942647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.942625    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:06:41.944028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:41.944002    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:06:43.246084 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:43.246049    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:43.246463 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:43.246271    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:48.348858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:48.330093    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:06:48.751869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:48.751001    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:06:48.751869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:48.751086    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:48.758853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:48.756946    5064 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:06:48.758853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:48.757013    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:50.051136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.051095    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:50.051618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.051370    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" podUID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" containerName="maas-api" containerID="cri-o://e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2" gracePeriod=30
Oct 30 15:06:50.586386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.586355    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:50.625252 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:50.625210    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:50.675043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.675001    5064 generic.go:356] "Generic (PLEG): container finished" podID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" containerID="e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2" exitCode=0
Oct 30 15:06:50.675241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.675064    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" event={"ID":"b25f5dea-9b81-4cea-a2bf-587c65120a9c","Type":"ContainerDied","Data":"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"}
Oct 30 15:06:50.675241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.675149    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl" event={"ID":"b25f5dea-9b81-4cea-a2bf-587c65120a9c","Type":"ContainerDied","Data":"d779d979f7a58e3e53073bdd03a30af4d8b2eecd80cc076a4c875da452786102"}
Oct 30 15:06:50.675241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.675183    5064 scope.go:122] "RemoveContainer" containerID="e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"
Oct 30 15:06:50.675241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.675221    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"
Oct 30 15:06:50.676499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.676472    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\" (UniqueName: \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\") pod \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") "
Oct 30 15:06:50.676852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.676819    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\") pod \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") "
Oct 30 15:06:50.677203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.677175    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") pod \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") "
Oct 30 15:06:50.677316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.677282    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\") pod \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\" (UID: \"b25f5dea-9b81-4cea-a2bf-587c65120a9c\") "
Oct 30 15:06:50.678185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.678146    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca" pod "b25f5dea-9b81-4cea-a2bf-587c65120a9c" (UID: "b25f5dea-9b81-4cea-a2bf-587c65120a9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:06:50.687963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.680602    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls" pod "b25f5dea-9b81-4cea-a2bf-587c65120a9c" (UID: "b25f5dea-9b81-4cea-a2bf-587c65120a9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:06:50.687963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.682327    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs" pod "b25f5dea-9b81-4cea-a2bf-587c65120a9c" (UID: "b25f5dea-9b81-4cea-a2bf-587c65120a9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:06:50.687963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.686344    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm" pod "b25f5dea-9b81-4cea-a2bf-587c65120a9c" (UID: "b25f5dea-9b81-4cea-a2bf-587c65120a9c"). InnerVolumeSpecName "kube-api-access-t4jqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:50.720451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.720390    5064 scope.go:122] "RemoveContainer" containerID="e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"
Oct 30 15:06:50.720779 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:50.720752    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2\": container with ID starting with e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2 not found: ID does not exist" containerID="e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"
Oct 30 15:06:50.720960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.720789    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2"} err="failed to get container status \"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2\": rpc error: code = NotFound desc = could not find container \"e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2\": container with ID starting with e13f7b143bd1365e615fcba65175a25a7985f0a6315a5ecdff105b4c72471cb2 not found: ID does not exist"
Oct 30 15:06:50.780276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.780232    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:50.780276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.780258    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25f5dea-9b81-4cea-a2bf-587c65120a9c-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:50.780276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.780273    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jqm\" (UniqueName: \"kubernetes.io/projected/b25f5dea-9b81-4cea-a2bf-587c65120a9c-kube-api-access-t4jqm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:50.780276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:50.780284    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25f5dea-9b81-4cea-a2bf-587c65120a9c-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:51.098960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:51.098921    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:06:51.107933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:51.099872    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" containerID="cri-o://da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940" gracePeriod=30
Oct 30 15:06:51.112009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:51.111970    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:51.126090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:51.126038    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c4cd83d6-7cb648f8fd-bljdl"]
Oct 30 15:06:51.272844 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:51.272795    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:06:52.270439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.270403    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:06:52.271497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271477    5064 cpu_manager.go:414] "containerMap: removing container" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:52.271497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271498    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:52.271697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271534    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" containerName="maas-api"
Oct 30 15:06:52.271697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271542    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" containerName="maas-api"
Oct 30 15:06:52.271802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271718    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:52.271802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.271736    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" containerName="maas-api"
Oct 30 15:06:52.276241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.276218    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.298930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.298900    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:06:52.303651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.303622    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15214ce4-openshift-default-dockercfg-42fqc\""
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.434951    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435023    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435162    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6c5\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435355    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435408    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435440    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435472    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435543    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.435856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.435594    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.546569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.546439    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.546921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.546876    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.547219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.547176    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.552723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.552410    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.567898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.552326    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.569576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.569119    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6c5\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.570132 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.570108    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.572854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.572594    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.573098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.573073    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.585182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.582163    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.585517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.584713    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.585776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.585061    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.586552 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.585972    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.592858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.592055    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.608253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.608216    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6c5\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.623883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.623846    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.625779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.625751    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.628747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.628706    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\") pod \"e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") " pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:52.844922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.830040    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25f5dea-9b81-4cea-a2bf-587c65120a9c" path="/var/lib/kubelet/pods/b25f5dea-9b81-4cea-a2bf-587c65120a9c/volumes"
Oct 30 15:06:52.898458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:52.898406    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:53.399793 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.399755    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:06:53.496580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.482230    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:53.496580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.482646    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" podUID="f368bc6a-b67a-405e-a572-a88ff29deea3" containerName="maas-api" containerID="cri-o://0bdaa1279f78fc3a6c84b8aa25b2aad7c1da4a3d9cedc6da6256620cd2bfc4f1" gracePeriod=30
Oct 30 15:06:53.728453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.686178    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:53.728453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.686279    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:53.728453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.686320    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:06:53.983918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:53.983809    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerStarted","Data":"75cad5685d6c3a6d2547fa0e810f58257bc9e44dac5fb861b3ca57c37745162e"}
Oct 30 15:06:54.170857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.165959    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:06:54.215622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.187670    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" containerID="cri-o://6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0" gracePeriod=30
Oct 30 15:06:54.997142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.997106    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerStarted","Data":"74fbbc6841edb57f04aea8088773d960d1551b075bc1e4aeab59471d2ef7ddc3"}
Oct 30 15:06:54.999384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.999356    5064 generic.go:356] "Generic (PLEG): container finished" podID="f368bc6a-b67a-405e-a572-a88ff29deea3" containerID="0bdaa1279f78fc3a6c84b8aa25b2aad7c1da4a3d9cedc6da6256620cd2bfc4f1" exitCode=0
Oct 30 15:06:54.999521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.999427    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" event={"ID":"f368bc6a-b67a-405e-a572-a88ff29deea3","Type":"ContainerDied","Data":"0bdaa1279f78fc3a6c84b8aa25b2aad7c1da4a3d9cedc6da6256620cd2bfc4f1"}
Oct 30 15:06:54.999521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.999469    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw" event={"ID":"f368bc6a-b67a-405e-a572-a88ff29deea3","Type":"ContainerDied","Data":"fdc57fed16b783e025974de969ef57bc006a895558e105e64184b6ed1e1faecc"}
Oct 30 15:06:54.999521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:54.999486    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc57fed16b783e025974de969ef57bc006a895558e105e64184b6ed1e1faecc"
Oct 30 15:06:55.011969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.011944    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:55.109687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.109569    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\") pod \"f368bc6a-b67a-405e-a572-a88ff29deea3\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") "
Oct 30 15:06:55.110459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.110429    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\") pod \"f368bc6a-b67a-405e-a572-a88ff29deea3\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") "
Oct 30 15:06:55.113212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.113162    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca" pod "f368bc6a-b67a-405e-a572-a88ff29deea3" (UID: "f368bc6a-b67a-405e-a572-a88ff29deea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.115015    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls" pod "f368bc6a-b67a-405e-a572-a88ff29deea3" (UID: "f368bc6a-b67a-405e-a572-a88ff29deea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.115270    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\" (UniqueName: \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\") pod \"f368bc6a-b67a-405e-a572-a88ff29deea3\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") "
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.115354    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\") pod \"f368bc6a-b67a-405e-a572-a88ff29deea3\" (UID: \"f368bc6a-b67a-405e-a572-a88ff29deea3\") "
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.116287    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.116307    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f368bc6a-b67a-405e-a572-a88ff29deea3-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.118306    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs" pod "f368bc6a-b67a-405e-a572-a88ff29deea3" (UID: "f368bc6a-b67a-405e-a572-a88ff29deea3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:06:55.119592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.118418    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs" pod "f368bc6a-b67a-405e-a572-a88ff29deea3" (UID: "f368bc6a-b67a-405e-a572-a88ff29deea3"). InnerVolumeSpecName "kube-api-access-b97bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:55.120141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.120100    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podStartSLOduration=3.120089193 podStartE2EDuration="3.120089193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:55.106045713 +0000 UTC m=+1164.547936434" watchObservedRunningTime="2026-10-30 15:06:55.120089193 +0000 UTC m=+1164.561979910"
Oct 30 15:06:55.217701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.217657    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97bs\" (UniqueName: \"kubernetes.io/projected/f368bc6a-b67a-405e-a572-a88ff29deea3-kube-api-access-b97bs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:55.217701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.217696    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f368bc6a-b67a-405e-a572-a88ff29deea3-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:55.266204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.266169    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:06:55.266452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.266425    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" containerID="cri-o://f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67" gracePeriod=30
Oct 30 15:06:55.899298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.899261    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:55.905123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:55.905096    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:56.008016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.007984    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"
Oct 30 15:06:56.008560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.008214    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:56.009741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.009718    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:06:56.117138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.117104    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:56.121572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.121544    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3834be83-5f99c5d4ff-6glkw"]
Oct 30 15:06:56.727571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.727125    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f368bc6a-b67a-405e-a572-a88ff29deea3" path="/var/lib/kubelet/pods/f368bc6a-b67a-405e-a572-a88ff29deea3/volumes"
Oct 30 15:06:56.820091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.820064    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:56.842991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.842950    5064 patch_prober.go:29] interesting pod/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:06:56.843169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.843010    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:06:56.863824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.863792    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.863950    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.863982    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864043    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864088    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864121    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864139    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:56.864326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864241    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864284    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.864436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864414    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\") pod \"0dc70a4a-4f39-4be1-a810-366401cd1254\" (UID: \"0dc70a4a-4f39-4be1-a810-366401cd1254\") "
Oct 30 15:06:56.865138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864582    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:06:56.865138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864709    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:56.865138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864775    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:56.865138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.864961    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:56.865422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.865313    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.865422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.865337    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.865422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.865355    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.865422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.865369    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.865422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.865383    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc70a4a-4f39-4be1-a810-366401cd1254-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.866612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.866584    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:06:56.867685 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.867658    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:06:56.869110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.869087    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "kube-api-access-kmhr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:56.869319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.869295    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token" pod "0dc70a4a-4f39-4be1-a810-366401cd1254" (UID: "0dc70a4a-4f39-4be1-a810-366401cd1254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:06:56.945490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.945390    5064 patch_prober.go:29] interesting pod/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 15:06:56.945490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.945460    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:06:56.966930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.966892    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.966930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.966930    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.967150 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.966946    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmhr5\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-kube-api-access-kmhr5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:56.967150 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:56.966959    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc70a4a-4f39-4be1-a810-366401cd1254-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:06:57.018927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.018892    5064 generic.go:356] "Generic (PLEG): container finished" podID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerID="da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940" exitCode=0
Oct 30 15:06:57.019399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.018953    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"
Oct 30 15:06:57.019399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.018977    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerDied","Data":"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"}
Oct 30 15:06:57.019399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.019050    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" event={"ID":"0dc70a4a-4f39-4be1-a810-366401cd1254","Type":"ContainerDied","Data":"f68d7e6692bc5447ca37a0b7606e1fd9654ff49821d1da159a6e9f13437fb97e"}
Oct 30 15:06:57.019399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.019086    5064 scope.go:122] "RemoveContainer" containerID="da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"
Oct 30 15:06:57.034025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.034002    5064 scope.go:122] "RemoveContainer" containerID="161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"
Oct 30 15:06:57.044991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.044970    5064 scope.go:122] "RemoveContainer" containerID="da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"
Oct 30 15:06:57.045273 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:57.045252    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940\": container with ID starting with da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940 not found: ID does not exist" containerID="da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"
Oct 30 15:06:57.045318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.045285    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940"} err="failed to get container status \"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940\": rpc error: code = NotFound desc = could not find container \"da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940\": container with ID starting with da00ed5fdf58e7dae710842c611030881798f523479b4a5593961fb52b9c1940 not found: ID does not exist"
Oct 30 15:06:57.045318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.045306    5064 scope.go:122] "RemoveContainer" containerID="161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"
Oct 30 15:06:57.045543 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:57.045526    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157\": container with ID starting with 161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157 not found: ID does not exist" containerID="161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"
Oct 30 15:06:57.045588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.045548    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157"} err="failed to get container status \"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157\": rpc error: code = NotFound desc = could not find container \"161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157\": container with ID starting with 161c95abd76e9810f5fafc5b35c70944d13ea58ff71b7969a91058ea65bfa157 not found: ID does not exist"
Oct 30 15:06:57.081667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.081634    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:06:57.086661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.086634    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj"]
Oct 30 15:06:57.803057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.803022    5064 patch_prober.go:29] interesting pod/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:06:57.803209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:57.803082    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c4cd83d6-gw-openshift-default-69bb5b895c-wjpmj" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:06:58.724405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:58.724368    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" path="/var/lib/kubelet/pods/0dc70a4a-4f39-4be1-a810-366401cd1254/volumes"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.543415    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544589    5064 cpu_manager.go:414] "containerMap: removing container" podUID="f368bc6a-b67a-405e-a572-a88ff29deea3" containerName="maas-api"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544607    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f368bc6a-b67a-405e-a572-a88ff29deea3" containerName="maas-api"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544633    5064 cpu_manager.go:414] "containerMap: removing container" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544642    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544664    5064 cpu_manager.go:414] "containerMap: removing container" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544674    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544712    5064 cpu_manager.go:414] "containerMap: removing container" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544719    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544882    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544913    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="f368bc6a-b67a-405e-a572-a88ff29deea3" containerName="maas-api"
Oct 30 15:06:59.549188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.544926    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa484c1-ef08-491e-a989-41113905658b" containerName="istio-proxy"
Oct 30 15:06:59.563244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.563213    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:06:59.568542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.568437    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.602533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.602489    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-985bf6-openshift-default-dockercfg-wt6nx\""
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.774439    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.774530    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.774635    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.774663    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.774820    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.777074    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj75\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.777141    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.777199    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.780856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.777225    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.850071 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:06:59.811412    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885463    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885512    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885699    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885751    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj75\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885793    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885843    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.885869    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.886003    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.886074    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.886615    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.886874    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.887654    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.888340    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.906855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.888592    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.978595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.952958    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.978595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.967558    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.978595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.969864    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:06:59.978595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:06:59.970679    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj75\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:00.011999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.011969    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:07:00.093825 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.093786    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.093889    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.093931    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.093960    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094029    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094082    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094203    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094329    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094371    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\") pod \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\" (UID: \"83ee1a63-d18b-4de0-b303-a73e06fba4a4\") "
Oct 30 15:07:00.094626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094591    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:07:00.094780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.094751    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.095377    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.095397    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.095404    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.095610    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.095983    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.116921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.097456    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.119076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.119002    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:07:00.132474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.120419    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "kube-api-access-f5db6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:00.154909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.134542    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token" pod "83ee1a63-d18b-4de0-b303-a73e06fba4a4" (UID: "83ee1a63-d18b-4de0-b303-a73e06fba4a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200092    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200131    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200144    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5db6\" (UniqueName: \"kubernetes.io/projected/83ee1a63-d18b-4de0-b303-a73e06fba4a4-kube-api-access-f5db6\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200161    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200173    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200184    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.200198    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ee1a63-d18b-4de0-b303-a73e06fba4a4-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.213195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.212676    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:00.383873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.381701    5064 generic.go:356] "Generic (PLEG): container finished" podID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerID="6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0" exitCode=0
Oct 30 15:07:00.383873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.382047    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerDied","Data":"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"}
Oct 30 15:07:00.383873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.382110    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd" event={"ID":"83ee1a63-d18b-4de0-b303-a73e06fba4a4","Type":"ContainerDied","Data":"b08503d525498ba3126537a2e2e4adb084cfa2591f454189b0f75f839794a45b"}
Oct 30 15:07:00.383873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.382141    5064 scope.go:122] "RemoveContainer" containerID="6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"
Oct 30 15:07:00.383873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.382267    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"
Oct 30 15:07:00.446754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.446726    5064 scope.go:122] "RemoveContainer" containerID="16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"
Oct 30 15:07:00.459510 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.459483    5064 scope.go:122] "RemoveContainer" containerID="6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"
Oct 30 15:07:00.459873 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:00.459814    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0\": container with ID starting with 6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0 not found: ID does not exist" containerID="6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"
Oct 30 15:07:00.459955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.459874    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0"} err="failed to get container status \"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0\": rpc error: code = NotFound desc = could not find container \"6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0\": container with ID starting with 6ddcd5ba7ef1d32fc13ced149ed3cf2aa647e2084a66055438f9d612c66eadc0 not found: ID does not exist"
Oct 30 15:07:00.459955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.459899    5064 scope.go:122] "RemoveContainer" containerID="16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"
Oct 30 15:07:00.460152 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:00.460133    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051\": container with ID starting with 16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051 not found: ID does not exist" containerID="16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"
Oct 30 15:07:00.460222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.460157    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051"} err="failed to get container status \"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051\": rpc error: code = NotFound desc = could not find container \"16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051\": container with ID starting with 16cad42f9f586863e4a42156420476ad6b6915c42c39692cec7faac8e4b0c051 not found: ID does not exist"
Oct 30 15:07:00.473125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.473098    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:07:00.483504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.483475    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dc34df04-openshift-default-6cfdf5df97-dkbdd"]
Oct 30 15:07:00.674214 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:07:00.674182    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bb4726_db4d_42f9_abd4_8bfcd6af938a.slice/crio-c8faa17c82250ec1a433a5982aa0feebe1cdb552c967a127d5158419debddfce WatchSource:0}: Error finding container c8faa17c82250ec1a433a5982aa0feebe1cdb552c967a127d5158419debddfce: Status 404 returned error can't find the container with id c8faa17c82250ec1a433a5982aa0feebe1cdb552c967a127d5158419debddfce
Oct 30 15:07:00.677395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.677353    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:07:00.679912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.678616    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:07:00.685798 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.685752    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:00.685943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.685914    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:00.686042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.686021    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:00.725163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.725125    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" path="/var/lib/kubelet/pods/83ee1a63-d18b-4de0-b303-a73e06fba4a4/volumes"
Oct 30 15:07:00.822723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.822687    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.822794    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.822862    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.822936    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.822988    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.823029    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.823042    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.823125    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.823199    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.823330    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\") pod \"62ac1230-a258-4779-ba13-6e23970da19a\" (UID: \"62ac1230-a258-4779-ba13-6e23970da19a\") "
Oct 30 15:07:00.824245 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.824198    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.824300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.824251    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.824300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.824280    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.824415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.824320    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.824415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.824345    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:07:00.826165 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.826129    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:07:00.826236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.826211    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:00.826902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.826878    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:00.827000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.826976    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl" pod "62ac1230-a258-4779-ba13-6e23970da19a" (UID: "62ac1230-a258-4779-ba13-6e23970da19a"). InnerVolumeSpecName "kube-api-access-8j8kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:00.926434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926339    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926388    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ac1230-a258-4779-ba13-6e23970da19a-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926402    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926416    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926429    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926440    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ac1230-a258-4779-ba13-6e23970da19a-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926453    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ac1230-a258-4779-ba13-6e23970da19a-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:00.926803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:00.926467    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8kl\" (UniqueName: \"kubernetes.io/projected/62ac1230-a258-4779-ba13-6e23970da19a-kube-api-access-8j8kl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:01.408919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.408825    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerStarted","Data":"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"}
Oct 30 15:07:01.409124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.408949    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerStarted","Data":"c8faa17c82250ec1a433a5982aa0feebe1cdb552c967a127d5158419debddfce"}
Oct 30 15:07:01.411804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.411778    5064 generic.go:356] "Generic (PLEG): container finished" podID="62ac1230-a258-4779-ba13-6e23970da19a" containerID="f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67" exitCode=0
Oct 30 15:07:01.411949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.411857    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerDied","Data":"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"}
Oct 30 15:07:01.411949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.411925    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm" event={"ID":"62ac1230-a258-4779-ba13-6e23970da19a","Type":"ContainerDied","Data":"f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56"}
Oct 30 15:07:01.412076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.411868    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"
Oct 30 15:07:01.412076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.411954    5064 scope.go:122] "RemoveContainer" containerID="f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"
Oct 30 15:07:01.424168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.424145    5064 scope.go:122] "RemoveContainer" containerID="75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"
Oct 30 15:07:01.437379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.437345    5064 scope.go:122] "RemoveContainer" containerID="f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"
Oct 30 15:07:01.437689 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:01.437664    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67\": container with ID starting with f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67 not found: ID does not exist" containerID="f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"
Oct 30 15:07:01.437771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.437694    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67"} err="failed to get container status \"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67\": rpc error: code = NotFound desc = could not find container \"f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67\": container with ID starting with f1efff912a6228077c38f2231f03a893e1b97d04c3b92a1a8246f318ede82e67 not found: ID does not exist"
Oct 30 15:07:01.437771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.437714    5064 scope.go:122] "RemoveContainer" containerID="75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"
Oct 30 15:07:01.437977 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:01.437952    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823\": container with ID starting with 75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823 not found: ID does not exist" containerID="75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"
Oct 30 15:07:01.438034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.437989    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823"} err="failed to get container status \"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823\": rpc error: code = NotFound desc = could not find container \"75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823\": container with ID starting with 75897d81a6d3198a493876df993b0b187c5b15df4f2121b2bd81de3d49367823 not found: ID does not exist"
Oct 30 15:07:01.461695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.461643    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podStartSLOduration=2.461629487 podStartE2EDuration="2.461629487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:01.458490212 +0000 UTC m=+1170.900380930" watchObservedRunningTime="2026-10-30 15:07:01.461629487 +0000 UTC m=+1170.903520203"
Oct 30 15:07:01.472758 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:01.472726    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ac1230_a258_4779_ba13_6e23970da19a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ac1230_a258_4779_ba13_6e23970da19a.slice/crio-f3c3bd0c0bcd68c7b306f03a5bffbddb05ef01d3beffce4de50d2dcc7f7f2d56\": RecentStats: unable to find data in memory cache]"
Oct 30 15:07:01.503281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.503247    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:07:01.506918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:01.506895    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3834be83-openshift-default-69dbd47d89-m25mm"]
Oct 30 15:07:02.213948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:02.213917    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:02.219716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:02.219690    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:02.422616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:02.422579    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:02.423606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:02.423580    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:07:02.725450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:02.725415    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ac1230-a258-4779-ba13-6e23970da19a" path="/var/lib/kubelet/pods/62ac1230-a258-4779-ba13-6e23970da19a/volumes"
Oct 30 15:07:05.299678 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:05.299637    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:07:11.641082 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:11.641006    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:07:13.057941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.057900    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:13.058639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058624    5064 cpu_manager.go:414] "containerMap: removing container" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:13.058695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058640    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:13.058695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058655    5064 cpu_manager.go:414] "containerMap: removing container" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:13.058695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058660    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:13.058695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058675    5064 cpu_manager.go:414] "containerMap: removing container" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:13.058695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058680    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:13.058878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058808    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:13.058878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058817    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc70a4a-4f39-4be1-a810-366401cd1254" containerName="istio-proxy"
Oct 30 15:07:13.058878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.058847    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:13.062785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.062765    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.085536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.085502    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:13.089441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.089418    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15214ce4-serving-cert\""
Oct 30 15:07:13.200186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.200146    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.200369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.200204    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.200470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.200447    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlvt\" (UniqueName: \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.200521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.200502    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.326906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.313896    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.326906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.314070    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.326906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.314138    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.338851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.328677    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlvt\" (UniqueName: \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.338851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.329752    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.345095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.339794    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.354820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.354743    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.379927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.365140    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlvt\" (UniqueName: \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\") pod \"maas-api-e2e-ait-15214ce4-74767467ff-rjhd7\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.380516 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.380478    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:13.741004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:13.740976    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:13.742534 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:07:13.742506    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda217940e_e9f1_4fdf_ac52_6a486bfcf11c.slice/crio-d95d568f2f55974be85076b9e36d20c64248bd68166249be50a2c59485f9d83c WatchSource:0}: Error finding container d95d568f2f55974be85076b9e36d20c64248bd68166249be50a2c59485f9d83c: Status 404 returned error can't find the container with id d95d568f2f55974be85076b9e36d20c64248bd68166249be50a2c59485f9d83c
Oct 30 15:07:14.601026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:14.600934    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" event={"ID":"a217940e-e9f1-4fdf-ac52-6a486bfcf11c","Type":"ContainerStarted","Data":"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"}
Oct 30 15:07:14.601026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:14.601000    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" event={"ID":"a217940e-e9f1-4fdf-ac52-6a486bfcf11c","Type":"ContainerStarted","Data":"d95d568f2f55974be85076b9e36d20c64248bd68166249be50a2c59485f9d83c"}
Oct 30 15:07:14.601451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:14.601039    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:14.700269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:14.700187    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" podStartSLOduration=1.301242115 podStartE2EDuration="1.70016084s" totalImagesPullingTime="398.918725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:14.683334829 +0000 UTC m=+1184.125225549" watchObservedRunningTime="2026-10-30 15:07:14.70016084 +0000 UTC m=+1184.142051557"
Oct 30 15:07:20.332581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.332539    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:07:20.333337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.333317    5064 cpu_manager.go:414] "containerMap: removing container" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:20.333337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.333335    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:20.333514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.333443    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ee1a63-d18b-4de0-b303-a73e06fba4a4" containerName="istio-proxy"
Oct 30 15:07:20.333514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.333459    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ac1230-a258-4779-ba13-6e23970da19a" containerName="istio-proxy"
Oct 30 15:07:20.337519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.337490    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.361111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.361057    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:07:20.368448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.368416    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-985bf6-serving-cert\""
Oct 30 15:07:20.379194 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:20.379160    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:07:20.457210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.457170    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.457391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.457322    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.457618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.457595    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cgm\" (UniqueName: \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.457732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.457711    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.559712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.559667    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cgm\" (UniqueName: \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.559921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.559749    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.559921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.559797    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.560074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.559927    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.560663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.560641    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.562542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.562517    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.562542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.562529    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.584941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.584818    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cgm\" (UniqueName: \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\") pod \"maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.671857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.671121    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:20.679856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:20.677986    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:21.117124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:21.117097    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:07:21.118540 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:07:21.118506    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c51e121_fb24_4916_b915_a2878e2686b6.slice/crio-e0401fa9be53fe5a18d2217c42dac725ec08577e817e913beeb5d47ad552ea5c WatchSource:0}: Error finding container e0401fa9be53fe5a18d2217c42dac725ec08577e817e913beeb5d47ad552ea5c: Status 404 returned error can't find the container with id e0401fa9be53fe5a18d2217c42dac725ec08577e817e913beeb5d47ad552ea5c
Oct 30 15:07:21.724504 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:21.720982    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa484c1_ef08_491e_a989_41113905658b.slice/crio-conmon-f48247ef0364aae406c70020890c802917f61de7e71350a042b0689eaede5fe2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 15:07:21.759958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:21.754970    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" event={"ID":"3c51e121-fb24-4916-b915-a2878e2686b6","Type":"ContainerStarted","Data":"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"}
Oct 30 15:07:21.759958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:21.755040    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" event={"ID":"3c51e121-fb24-4916-b915-a2878e2686b6","Type":"ContainerStarted","Data":"e0401fa9be53fe5a18d2217c42dac725ec08577e817e913beeb5d47ad552ea5c"}
Oct 30 15:07:21.759958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:21.756211    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:21.815899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:21.815765    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" podStartSLOduration=1.409254789 podStartE2EDuration="1.81574684s" totalImagesPullingTime="406.492051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:21.814909103 +0000 UTC m=+1191.256799884" watchObservedRunningTime="2026-10-30 15:07:21.81574684 +0000 UTC m=+1191.257637580"
Oct 30 15:07:28.858049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:28.857992    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:07:29.651854 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:29.651811    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:07:30.878846 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:30.878793    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:07:30.879351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:30.879132    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:07:38.941291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.941253    5064 generic.go:356] "Generic (PLEG): container finished" podID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerID="74fbbc6841edb57f04aea8088773d960d1551b075bc1e4aeab59471d2ef7ddc3" exitCode=0
Oct 30 15:07:38.941861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.941317    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerDied","Data":"74fbbc6841edb57f04aea8088773d960d1551b075bc1e4aeab59471d2ef7ddc3"}
Oct 30 15:07:38.941966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.941946    5064 scope.go:122] "RemoveContainer" containerID="74fbbc6841edb57f04aea8088773d960d1551b075bc1e4aeab59471d2ef7ddc3"
Oct 30 15:07:38.943577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.943547    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:38.943647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.943618    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:38.943684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:38.943653    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:39.952295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:39.952254    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerStarted","Data":"d8e2d1b6545df65ae9685d413435e0d87d45610203d1154537b0e847cb4b7931"}
Oct 30 15:07:40.900874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:40.900769    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:07:40.902027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:40.901997    5064 patch_prober.go:29] interesting pod/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 15:07:40.902268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:40.902242    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 15:07:41.008391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:41.008357    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:07:41.912555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:41.912518    5064 patch_prober.go:29] interesting pod/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 15:07:41.912856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:41.912802    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 15:07:42.899406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:42.899373    5064 patch_prober.go:29] interesting pod/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 15:07:42.899795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:42.899425    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 15:07:43.900638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:43.900601    5064 patch_prober.go:29] interesting pod/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 15:07:43.901086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:43.900662    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 15:07:44.903851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:44.901552    5064 patch_prober.go:29] interesting pod/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 15:07:44.903851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:44.902166    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 15:07:45.747887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.747847    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:07:45.752611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.752581    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.784223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.784180    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:07:45.803390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.803355    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-609709-openshift-default-dockercfg-78zbk\""
Oct 30 15:07:45.888253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888213    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888281    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888430    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888464    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888600    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888653    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888876    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.888937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.888932    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwvc\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.889071 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.889012    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:45.905022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.904990    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:07:45.906313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:45.906287    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006001    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006078    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwvc\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006132    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006214    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006271    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006357    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006495    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006568    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.006910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.006622    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.017821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.017782    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.019091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.019064    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.019473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.019450    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.038853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.033201    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.039748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.039687    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.040651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.040624    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.097733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.097675    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.189822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.189743    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwvc\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.193761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.191253    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.447480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.447180    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:46.839811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.839781    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:07:46.840263 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:07:46.840230    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0259dc_47bb_41a7_976a_3d56d504bfdc.slice/crio-2db0d516f0d2da5db9f906eb752794c431bda11d79e77f9e34acfefdc4eec28f WatchSource:0}: Error finding container 2db0d516f0d2da5db9f906eb752794c431bda11d79e77f9e34acfefdc4eec28f: Status 404 returned error can't find the container with id 2db0d516f0d2da5db9f906eb752794c431bda11d79e77f9e34acfefdc4eec28f
Oct 30 15:07:46.843365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.843324    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:46.843469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.843453    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:46.843519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:46.843500    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:47.198562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:47.198516    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerStarted","Data":"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"}
Oct 30 15:07:47.199056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:47.198592    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerStarted","Data":"2db0d516f0d2da5db9f906eb752794c431bda11d79e77f9e34acfefdc4eec28f"}
Oct 30 15:07:47.261649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:47.261595    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podStartSLOduration=2.2615803039999998 podStartE2EDuration="2.261580304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:47.261463922 +0000 UTC m=+1216.703354640" watchObservedRunningTime="2026-10-30 15:07:47.261580304 +0000 UTC m=+1216.703471021"
Oct 30 15:07:47.464323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:47.464184    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:47.483524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:47.483486    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:48.206361 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:48.206330    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:48.207381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:48.207358    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:07:51.547586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:51.547549    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:51.548234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:51.547779    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" podUID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" containerName="maas-api" containerID="cri-o://72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc" gracePeriod=30
Oct 30 15:07:52.200762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.200727    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:52.304854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.300916    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\" (UniqueName: \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\") pod \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") "
Oct 30 15:07:52.304854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.301135    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\") pod \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") "
Oct 30 15:07:52.304854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.301357    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\") pod \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") "
Oct 30 15:07:52.304854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.301462    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\") pod \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\" (UID: \"a217940e-e9f1-4fdf-ac52-6a486bfcf11c\") "
Oct 30 15:07:52.304854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.303739    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca" pod "a217940e-e9f1-4fdf-ac52-6a486bfcf11c" (UID: "a217940e-e9f1-4fdf-ac52-6a486bfcf11c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:07:52.338190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.336321    5064 generic.go:356] "Generic (PLEG): container finished" podID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" containerID="72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc" exitCode=0
Oct 30 15:07:52.338190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.336609    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" event={"ID":"a217940e-e9f1-4fdf-ac52-6a486bfcf11c","Type":"ContainerDied","Data":"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"}
Oct 30 15:07:52.338190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.336663    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7" event={"ID":"a217940e-e9f1-4fdf-ac52-6a486bfcf11c","Type":"ContainerDied","Data":"d95d568f2f55974be85076b9e36d20c64248bd68166249be50a2c59485f9d83c"}
Oct 30 15:07:52.338190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.336693    5064 scope.go:122] "RemoveContainer" containerID="72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"
Oct 30 15:07:52.339096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.338646    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"
Oct 30 15:07:52.351943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.351706    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls" pod "a217940e-e9f1-4fdf-ac52-6a486bfcf11c" (UID: "a217940e-e9f1-4fdf-ac52-6a486bfcf11c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:07:52.408241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.386196    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs" pod "a217940e-e9f1-4fdf-ac52-6a486bfcf11c" (UID: "a217940e-e9f1-4fdf-ac52-6a486bfcf11c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:07:52.427482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.408720    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:52.427482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.408752    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:52.427482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.408766    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:52.491465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.490640    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt" pod "a217940e-e9f1-4fdf-ac52-6a486bfcf11c" (UID: "a217940e-e9f1-4fdf-ac52-6a486bfcf11c"). InnerVolumeSpecName "kube-api-access-xvlvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:52.591430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.591337    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlvt\" (UniqueName: \"kubernetes.io/projected/a217940e-e9f1-4fdf-ac52-6a486bfcf11c-kube-api-access-xvlvt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:52.696070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.692424    5064 scope.go:122] "RemoveContainer" containerID="72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"
Oct 30 15:07:52.700475 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:52.700421    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc\": container with ID starting with 72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc not found: ID does not exist" containerID="72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"
Oct 30 15:07:52.700803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.700500    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc"} err="failed to get container status \"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc\": rpc error: code = NotFound desc = could not find container \"72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc\": container with ID starting with 72ac63e892f6b18c049265e0feae46ab619aa8342bc865c773f7a480671370bc not found: ID does not exist"
Oct 30 15:07:52.766299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.766269    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:52.774967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:52.774762    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15214ce4-74767467ff-rjhd7"]
Oct 30 15:07:54.218991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:54.218957    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:07:54.219795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:54.219762    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy" containerID="cri-o://d8e2d1b6545df65ae9685d413435e0d87d45610203d1154537b0e847cb4b7931" gracePeriod=30
Oct 30 15:07:54.724597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:54.724565    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" path="/var/lib/kubelet/pods/a217940e-e9f1-4fdf-ac52-6a486bfcf11c/volumes"
Oct 30 15:07:55.581907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.581873    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:07:55.582755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.582733    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" containerName="maas-api"
Oct 30 15:07:55.582755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.582757    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" containerName="maas-api"
Oct 30 15:07:55.583001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.582891    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a217940e-e9f1-4fdf-ac52-6a486bfcf11c" containerName="maas-api"
Oct 30 15:07:55.588271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.588244    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.623416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.623384    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-416d92-openshift-default-dockercfg-7l9zb\""
Oct 30 15:07:55.630235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.630200    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:07:55.647268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647231    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647292    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg66j\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647377    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647497    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647560    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647586    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647699    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.647821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.647729    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757410    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757490    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg66j\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757542    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757678    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757723    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757751    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757786    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757895    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.757928    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.758769    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.759077    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.760658    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.761248    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.765857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.761468    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.796769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.796693    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.797470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.797421    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.798214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.798167    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.800727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.800705    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg66j\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:55.911855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:55.911134    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:56.416247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:56.416213    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:07:56.416580 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:07:56.416553    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b564f0_6bd7_4ea3_b5e1_6c104a805240.slice/crio-e2b15527ebd8f0f3af5276cbcd8341bfe06f76d24ca89a2fb63e024ee5b993a6 WatchSource:0}: Error finding container e2b15527ebd8f0f3af5276cbcd8341bfe06f76d24ca89a2fb63e024ee5b993a6: Status 404 returned error can't find the container with id e2b15527ebd8f0f3af5276cbcd8341bfe06f76d24ca89a2fb63e024ee5b993a6
Oct 30 15:07:56.419434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:56.419398    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:56.419508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:56.419471    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:56.419508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:56.419501    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:07:57.378245 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:57.378208    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerStarted","Data":"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"}
Oct 30 15:07:57.378691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:57.378265    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerStarted","Data":"e2b15527ebd8f0f3af5276cbcd8341bfe06f76d24ca89a2fb63e024ee5b993a6"}
Oct 30 15:07:57.444887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:57.444808    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podStartSLOduration=2.44479182 podStartE2EDuration="2.44479182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:57.441218132 +0000 UTC m=+1226.883108850" watchObservedRunningTime="2026-10-30 15:07:57.44479182 +0000 UTC m=+1226.886682537"
Oct 30 15:07:57.912571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:57.912528    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:57.918893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:57.918864    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:58.385690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:58.385653    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:58.386714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:58.386690    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:07:59.396592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.396553    5064 generic.go:356] "Generic (PLEG): container finished" podID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerID="d8e2d1b6545df65ae9685d413435e0d87d45610203d1154537b0e847cb4b7931" exitCode=0
Oct 30 15:07:59.397096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.396620    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerDied","Data":"d8e2d1b6545df65ae9685d413435e0d87d45610203d1154537b0e847cb4b7931"}
Oct 30 15:07:59.397096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.396690    5064 scope.go:122] "RemoveContainer" containerID="74fbbc6841edb57f04aea8088773d960d1551b075bc1e4aeab59471d2ef7ddc3"
Oct 30 15:07:59.517500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.517468    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:07:59.652730 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:07:59.652697    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:07:59.686134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686103    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686156    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686274    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686335    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686368    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686384    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:59.686440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686409    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686463    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686507    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.686729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686679    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:07:59.686897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.686792    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\") pod \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\" (UID: \"64b6379c-99f2-403a-bbcb-fa004a5da9ba\") "
Oct 30 15:07:59.687318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687282    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:59.687485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687460    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:59.687969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687743    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.687969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687903    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.687969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687921    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.687969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687934    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.687969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.687869    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:59.688941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.688915    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:59.689140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.689108    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:07:59.689529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.689505    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:07:59.689607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.689531    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5" pod "64b6379c-99f2-403a-bbcb-fa004a5da9ba" (UID: "64b6379c-99f2-403a-bbcb-fa004a5da9ba"). InnerVolumeSpecName "kube-api-access-8n6c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:07:59.789655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.789564    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.789655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.789596    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.789655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.789605    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n6c5\" (UniqueName: \"kubernetes.io/projected/64b6379c-99f2-403a-bbcb-fa004a5da9ba-kube-api-access-8n6c5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.789655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.789616    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b6379c-99f2-403a-bbcb-fa004a5da9ba-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:07:59.789655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:07:59.789625    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b6379c-99f2-403a-bbcb-fa004a5da9ba-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:08:00.406238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.406198    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"
Oct 30 15:08:00.406708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.406197    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7" event={"ID":"64b6379c-99f2-403a-bbcb-fa004a5da9ba","Type":"ContainerDied","Data":"75cad5685d6c3a6d2547fa0e810f58257bc9e44dac5fb861b3ca57c37745162e"}
Oct 30 15:08:00.406708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.406358    5064 scope.go:122] "RemoveContainer" containerID="d8e2d1b6545df65ae9685d413435e0d87d45610203d1154537b0e847cb4b7931"
Oct 30 15:08:00.485009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.484969    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:08:00.500675 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.500637    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15214ce4-openshift-default-5f9b487b8c-zgql7"]
Oct 30 15:08:00.738446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:00.738362    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" path="/var/lib/kubelet/pods/64b6379c-99f2-403a-bbcb-fa004a5da9ba/volumes"
Oct 30 15:08:08.342756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.342709    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:08:08.343775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.343744    5064 cpu_manager.go:414] "containerMap: removing container" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.343775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.343774    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.343994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.343816    5064 cpu_manager.go:414] "containerMap: removing container" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.343994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.343822    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.344072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.344023    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.344072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.344042    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b6379c-99f2-403a-bbcb-fa004a5da9ba" containerName="istio-proxy"
Oct 30 15:08:08.349280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.349255    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.370052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.370024    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:08:08.382138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.382107    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-609709-serving-cert\""
Oct 30 15:08:08.420710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.420674    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.420710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.420723    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mf6\" (UniqueName: \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.421093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.421059    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.421259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.421238    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.523297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.523264    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.523483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.523378    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.523537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.523493    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.523537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.523530    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mf6\" (UniqueName: \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.524191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.524165    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.525869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.525820    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.525994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.525917    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.551781 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.551753    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mf6\" (UniqueName: \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\") pod \"maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:08.670566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:08.669206    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:09.086891 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:09.086857    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a233db_99b1_4dfb_91fb_5a818a3c4c1a.slice/crio-55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557 WatchSource:0}: Error finding container 55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557: Status 404 returned error can't find the container with id 55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557
Oct 30 15:08:09.088809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:09.088784    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:08:09.515700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:09.515669    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" event={"ID":"20a233db-99b1-4dfb-91fb-5a818a3c4c1a","Type":"ContainerStarted","Data":"55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557"}
Oct 30 15:08:10.535890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:10.535795    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" event={"ID":"20a233db-99b1-4dfb-91fb-5a818a3c4c1a","Type":"ContainerStarted","Data":"b756a80b856c4905dc1df27d20fb26b7f14c4de42ab00f39c25af175bb9fd58a"}
Oct 30 15:08:10.536720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:10.536695    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:10.584584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:10.584524    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" podStartSLOduration=2.211347507 podStartE2EDuration="2.584507406s" totalImagesPullingTime="373.159899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:10.582411095 +0000 UTC m=+1240.024301816" watchObservedRunningTime="2026-10-30 15:08:10.584507406 +0000 UTC m=+1240.026398160"
Oct 30 15:08:17.323094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.323050    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:08:17.328014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.327988    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.349145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.349117    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:08:17.362031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.361993    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-416d92-serving-cert\""
Oct 30 15:08:17.370470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.370436    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.370708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.370493    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.370708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.370539    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.370708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.370692    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrbg\" (UniqueName: \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.472454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.472413    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.472454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.472459    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.472683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.472583    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.472787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.472769    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrbg\" (UniqueName: \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.473239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.473209    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.474918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.474901    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.475061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.475040    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.498696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.498662    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrbg\" (UniqueName: \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\") pod \"maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:17.621884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.619115    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:08:17.654311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:17.644126    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:18.107005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.106968    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:08:18.107139 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:18.107040    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598bc32b_9092_41c0_a22d_826b7b937abc.slice/crio-091807d7b873fcef2ececeb96ba501993d3df0f2271ec50117ae1edcebed2e6e WatchSource:0}: Error finding container 091807d7b873fcef2ececeb96ba501993d3df0f2271ec50117ae1edcebed2e6e: Status 404 returned error can't find the container with id 091807d7b873fcef2ececeb96ba501993d3df0f2271ec50117ae1edcebed2e6e
Oct 30 15:08:18.579261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.576530    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.585593    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.587314    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.587582    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.587708    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.587744    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.587904    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.588134    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrbk\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.588174    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.588288    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.588564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.588403    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.608310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.607986    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:08:18.615882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.610245    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7acff45-openshift-default-dockercfg-qjdhh\""
Oct 30 15:08:18.656266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.656224    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" event={"ID":"598bc32b-9092-41c0-a22d-826b7b937abc","Type":"ContainerStarted","Data":"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"}
Oct 30 15:08:18.656475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.656290    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" event={"ID":"598bc32b-9092-41c0-a22d-826b7b937abc","Type":"ContainerStarted","Data":"091807d7b873fcef2ececeb96ba501993d3df0f2271ec50117ae1edcebed2e6e"}
Oct 30 15:08:18.656546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.656477    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:18.690445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690404    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690522    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690576    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690668    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690777    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrbk\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690809    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.690874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690853    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.691042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690873    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.691042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.690997    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.691162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.691088    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.691242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.691001    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.691387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.691247    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.692968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.691393    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.692968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.691894    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.693353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.693336    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.694160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.694141    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.708381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.708319    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" podStartSLOduration=1.285508315 podStartE2EDuration="1.708300883s" totalImagesPullingTime="422.792568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:18.702444039 +0000 UTC m=+1248.144334759" watchObservedRunningTime="2026-10-30 15:08:18.708300883 +0000 UTC m=+1248.150191601"
Oct 30 15:08:18.720772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.720740    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.721676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.721650    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrbk\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:18.926116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:18.923823    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:19.583316 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:19.583283    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c8c585_4cea_4bed_95fd_408cf3362a2c.slice/crio-efeada85cd486a2a1e50868c927b2edaf8afacc11b0c811dfa8f78e494f43663 WatchSource:0}: Error finding container efeada85cd486a2a1e50868c927b2edaf8afacc11b0c811dfa8f78e494f43663: Status 404 returned error can't find the container with id efeada85cd486a2a1e50868c927b2edaf8afacc11b0c811dfa8f78e494f43663
Oct 30 15:08:19.583638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:19.583456    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:08:19.586284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:19.586249    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:19.586357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:19.586318    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:19.586357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:19.586352    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:19.699086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:19.698953    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerStarted","Data":"efeada85cd486a2a1e50868c927b2edaf8afacc11b0c811dfa8f78e494f43663"}
Oct 30 15:08:20.707788 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:20.707743    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerStarted","Data":"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"}
Oct 30 15:08:20.787525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:20.787459    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podStartSLOduration=2.7874446109999997 podStartE2EDuration="2.787444611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:20.769775666 +0000 UTC m=+1250.211666383" watchObservedRunningTime="2026-10-30 15:08:20.787444611 +0000 UTC m=+1250.229335327"
Oct 30 15:08:20.931314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:20.931271    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:20.976791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:20.973490    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:21.716571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:21.716532    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:21.717721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:21.717699    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:08:22.361527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.361490    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:08:22.373264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.373227    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.389332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.389303    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:08:22.461507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461457    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461619    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461676    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461763    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461824    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461946    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.461995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.461980    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.462104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.462004    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhg4\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.462161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.462150    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563275    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563355    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563447    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563508    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563576    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563640    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563751    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563785    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563808    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563863    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563889    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.563946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.563923    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhg4\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.564220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.564197    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.564422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.564404    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.565806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.565779    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.566076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.566056    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.589318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.589291    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.589429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.589378    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhg4\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\") pod \"maas-default-gateway-openshift-default-6cc458d887-ssmkv\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:22.693122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:22.693091    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:23.028616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.028576    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:08:23.029322 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:23.029294    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb194ba01_7348_4500_a555_b2009e03d314.slice/crio-4fcbf10729453cd619262cd9892ad523757561f444bff5191618610e682d807d WatchSource:0}: Error finding container 4fcbf10729453cd619262cd9892ad523757561f444bff5191618610e682d807d: Status 404 returned error can't find the container with id 4fcbf10729453cd619262cd9892ad523757561f444bff5191618610e682d807d
Oct 30 15:08:23.032213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.032175    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:23.032318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.032260    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:23.032318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.032305    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:23.737281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.737240    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" event={"ID":"b194ba01-7348-4500-a555-b2009e03d314","Type":"ContainerStarted","Data":"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"}
Oct 30 15:08:23.737499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.737301    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" event={"ID":"b194ba01-7348-4500-a555-b2009e03d314","Type":"ContainerStarted","Data":"4fcbf10729453cd619262cd9892ad523757561f444bff5191618610e682d807d"}
Oct 30 15:08:23.803400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:23.803339    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podStartSLOduration=1.803316489 podStartE2EDuration="1.803316489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:23.800484894 +0000 UTC m=+1253.242375625" watchObservedRunningTime="2026-10-30 15:08:23.803316489 +0000 UTC m=+1253.245207206"
Oct 30 15:08:24.693860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:24.693800    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:24.696042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:24.696004    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:24.696175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:24.696064    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:24.704149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:24.704125    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:08:25.694915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:25.694858    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:25.695369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:25.694926    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:26.693500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:26.693467    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:26.693694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:26.693524    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:27.694135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:27.694099    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:27.694534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:27.694152    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:28.693823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:28.693784    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:28.694026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:28.693893    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:29.696347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:29.696307    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:29.697050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:29.697011    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:29.739089 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:08:29.739047    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:08:30.693694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:30.693653    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:30.693886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:30.693734    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:31.699362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:31.699325    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 15:08:31.700439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:31.700026    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 15:08:33.484314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:33.484257    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:33.486753 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:33.486614    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:33.487939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:33.487918    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:08:39.730123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.730086    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:08:39.778081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.778045    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:08:39.778276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.778147    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.796544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.796511    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d57911-openshift-default-dockercfg-v99pm\""
Oct 30 15:08:39.877232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877195    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877240    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877456    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877509    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877560    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877662    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2b9z\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877867    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.877992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877941    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.878039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.877997    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.982991    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983057    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983253    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983301    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983355    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983384    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2b9z\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983435    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983481    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.983550    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.984359    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.984628    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:39.987902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:39.984908    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.007859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.003206    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.007859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.003853    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.023856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.018911    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.041869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.038444    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.045850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.043255    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2b9z\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.045850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.045473    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.112857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.101789    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:40.675509 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:40.675432    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69852b89_3caa_4c13_a40a_8bb194876a0e.slice/crio-62b800060130b73e6f30fc5970d978d66e80f4a5a27502aa053e1fac9eaf87a4 WatchSource:0}: Error finding container 62b800060130b73e6f30fc5970d978d66e80f4a5a27502aa053e1fac9eaf87a4: Status 404 returned error can't find the container with id 62b800060130b73e6f30fc5970d978d66e80f4a5a27502aa053e1fac9eaf87a4
Oct 30 15:08:40.677176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.677142    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:08:40.679341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.679293    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:40.679433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.679388    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:40.679501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:40.679434    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:41.112199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:41.112110    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerStarted","Data":"8917048906aff05ef01bea6321a91643342a12ec82f5b5031cce20dd00695db2"}
Oct 30 15:08:41.112199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:41.112175    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerStarted","Data":"62b800060130b73e6f30fc5970d978d66e80f4a5a27502aa053e1fac9eaf87a4"}
Oct 30 15:08:41.176356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:41.176301    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podStartSLOduration=2.176285237 podStartE2EDuration="2.176285237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:41.173955033 +0000 UTC m=+1270.615845774" watchObservedRunningTime="2026-10-30 15:08:41.176285237 +0000 UTC m=+1270.618175955"
Oct 30 15:08:42.105409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.105287    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:42.151750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.150692    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:42.532621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.532579    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:08:42.570103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.569227    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:08:42.570103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.569358    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.591643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.591327    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-09e8eb-openshift-default-dockercfg-9727f\""
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599219    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599287    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599338    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599477    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599716    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksskg\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599761    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599796    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.600103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.599865    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.735863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.724871    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.768406    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksskg\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.768588    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.768729    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.768889    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.739046    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.769215    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.769390    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.769541    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.770035    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.776880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.771581    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.817918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.816583    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.923853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.883193    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:42.945851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:42.924874    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:43.001545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.001483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:43.102907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.102439    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:43.184997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.184790    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksskg\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:43.205875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.205750    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:43.207975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.207640    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:08:43.208259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.208222    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:43.510849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:43.505489    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:44.045257 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:44.044242    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dd2eeb_dd55_47b1_908b_1ded7a67517a.slice/crio-a13dbd2c0372581ccd7b9ac1bd6fdd8a739aa36f531bd82b9576445ff8884a34 WatchSource:0}: Error finding container a13dbd2c0372581ccd7b9ac1bd6fdd8a739aa36f531bd82b9576445ff8884a34: Status 404 returned error can't find the container with id a13dbd2c0372581ccd7b9ac1bd6fdd8a739aa36f531bd82b9576445ff8884a34
Oct 30 15:08:44.045257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.045161    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:08:44.048097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.048061    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:44.048240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.048143    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:44.048240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.048186    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:44.243499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.240233    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerStarted","Data":"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"}
Oct 30 15:08:44.243499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.240303    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerStarted","Data":"a13dbd2c0372581ccd7b9ac1bd6fdd8a739aa36f531bd82b9576445ff8884a34"}
Oct 30 15:08:44.300399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.300290    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podStartSLOduration=2.300274965 podStartE2EDuration="2.300274965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:44.296979482 +0000 UTC m=+1273.738870198" watchObservedRunningTime="2026-10-30 15:08:44.300274965 +0000 UTC m=+1273.742165681"
Oct 30 15:08:44.516012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:44.515976    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:45.557213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:45.557181    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.291903    5064 generic.go:356] "Generic (PLEG): container finished" podID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerID="fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3" exitCode=0
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.291978    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerDied","Data":"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"}
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.293535    5064 scope.go:122] "RemoveContainer" containerID="fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.296358    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.297613    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.299251    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.299348    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:46.301860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.299406    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:46.600814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.600728    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:08:46.606417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.606390    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.655145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.655022    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-100b2e-openshift-default-dockercfg-bsnrg\""
Oct 30 15:08:46.662967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.662929    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:08:46.677869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.677800    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.677944    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678005    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678081    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678165    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678198    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.678338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678283    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.679159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678634    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.679159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.678689    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkg5\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790071    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790138    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790168    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790234    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790476    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790505    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkg5\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790631    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790701    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.798336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.790738    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.826874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.825569    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.826874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.826277    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.826874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.826557    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.863986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.852143    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.885965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.884160    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.949869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.935483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.949869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.936069    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkg5\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.950896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.950795    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:46.966888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:46.966591    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:47.328284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:47.328214    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:47.527504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:47.527467    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerStarted","Data":"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"}
Oct 30 15:08:48.205912 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:08:48.205873    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852e8709_0822_4ce0_be18_a8ac4c8885f0.slice/crio-41fd83bda2118374128a98cccc10774525ef82ce5fe25e9497e5cc8c7e84c82d WatchSource:0}: Error finding container 41fd83bda2118374128a98cccc10774525ef82ce5fe25e9497e5cc8c7e84c82d: Status 404 returned error can't find the container with id 41fd83bda2118374128a98cccc10774525ef82ce5fe25e9497e5cc8c7e84c82d
Oct 30 15:08:48.206980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.206949    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:08:48.207665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.207636    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:08:48.209901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.209859    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:48.209987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.209955    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:48.210027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.209999    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:48.481800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.481710    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:08:48.484032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.482550    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:48.484032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.482610    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:48.620858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.601528    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerStarted","Data":"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"}
Oct 30 15:08:48.620858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.601626    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerStarted","Data":"41fd83bda2118374128a98cccc10774525ef82ce5fe25e9497e5cc8c7e84c82d"}
Oct 30 15:08:48.722908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:48.722686    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podStartSLOduration=2.722664951 podStartE2EDuration="2.722664951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:48.661384803 +0000 UTC m=+1278.103275523" watchObservedRunningTime="2026-10-30 15:08:48.722664951 +0000 UTC m=+1278.164555669"
Oct 30 15:08:49.348087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:49.347593    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:49.351813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:49.351789    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:08:49.352378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:49.352345    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:08:49.470864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:49.470091    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:49.470864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:49.470150    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:50.450174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:50.450138    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:50.464336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:50.463163    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:50.464336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:50.463222    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:50.619988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:50.619958    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:50.622306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:50.622011    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:08:51.507506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:51.505408    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:51.507506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:51.505512    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:52.461730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:52.461688    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:52.461933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:52.461749    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:53.475850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:53.472112    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:53.481870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:53.480848    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:54.462107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:54.462070    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 15:08:54.462296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:54.462127    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 15:08:55.466633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:55.466591    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:08:55.467760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:55.467737    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:08:56.925446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.925361    5064 generic.go:356] "Generic (PLEG): container finished" podID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerID="c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9" exitCode=0
Oct 30 15:08:56.925446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.925408    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerDied","Data":"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"}
Oct 30 15:08:56.925976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.925956    5064 scope.go:122] "RemoveContainer" containerID="c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"
Oct 30 15:08:56.927787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.927757    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:56.927903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.927868    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:56.927950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:56.927909    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:08:58.007373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:58.007325    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerStarted","Data":"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"}
Oct 30 15:08:58.385779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:58.385740    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:08:58.911894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:58.911818    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:08:58.935857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:58.932010    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:08:58.935857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:58.932082    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:08:59.767864 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:08:59.767576    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:08:59.912654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:59.912618    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:08:59.912867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:08:59.912679    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:09:00.912578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:00.912537    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:09:00.913037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:00.912598    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:09:01.913871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:01.912863    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:09:01.913871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:01.912949    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:09:02.943000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:02.942961    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:09:02.943572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:02.943541    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:09:03.935856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:03.935640    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 15:09:03.935856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:03.935706    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 15:09:04.557854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:04.532955    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ssmkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:09:04.557854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:04.533033    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:09:05.038186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:05.038150    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:09:05.081498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:05.081466    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:09:05.412362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:05.412295    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:09:05.915254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:05.856761    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:09:07.042855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.041808    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-mhmsw-5fr7v"]
Oct 30 15:09:07.047677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.047652    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.050483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.050452    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mhmsw-5fr7v"]
Oct 30 15:09:07.144674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.144636    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.144991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.144964    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr89q\" (UniqueName: \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.145091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.145070    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.145139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.145103    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.247059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.247003    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr89q\" (UniqueName: \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.247267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.247125    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.247267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.247151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.247267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.247188    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.247650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.247622    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.248082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.248059    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.248206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.248184    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.272371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.272341    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr89q\" (UniqueName: \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\") pod \"periodic-gathering-mhmsw-5fr7v\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") " pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.370118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.365951    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:09:07.888491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:07.888464    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mhmsw-5fr7v"]
Oct 30 15:09:07.888627 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:09:07.888556    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b224f_88bf_4956_a1dd_505bdb9a9e54.slice/crio-de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70 WatchSource:0}: Error finding container de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70: Status 404 returned error can't find the container with id de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70
Oct 30 15:09:08.271777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:08.271682    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v" event={"ID":"a41b224f-88bf-4956-a1dd-505bdb9a9e54","Type":"ContainerStarted","Data":"6775333de6cb4f9b6b6a0eca8cf8d0efcb8a02d54b6a630ce3aca1da85634cc2"}
Oct 30 15:09:08.271777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:08.271730    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v" event={"ID":"a41b224f-88bf-4956-a1dd-505bdb9a9e54","Type":"ContainerStarted","Data":"de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70"}
Oct 30 15:09:08.318053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:08.317988    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v" podStartSLOduration=2.317969084 podStartE2EDuration="2.317969084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:08.315165041 +0000 UTC m=+1297.757055758" watchObservedRunningTime="2026-10-30 15:09:08.317969084 +0000 UTC m=+1297.759859804"
Oct 30 15:09:13.311225 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.311187    5064 generic.go:356] "Generic (PLEG): container finished" podID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerID="650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25" exitCode=0
Oct 30 15:09:13.311789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.311250    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerDied","Data":"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"}
Oct 30 15:09:13.311923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.311902    5064 scope.go:122] "RemoveContainer" containerID="650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"
Oct 30 15:09:13.313943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.313908    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:13.314045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.313999    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:13.314089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:13.314046    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:14.351766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:14.346634    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerStarted","Data":"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"}
Oct 30 15:09:14.931854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:14.930249    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:09:14.931854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:14.930380    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:14.931854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:14.930425    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:15.933346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:15.933301    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:15.934023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:15.933987    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:16.926299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:16.926260    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:16.926574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:16.926544    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:17.931855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:17.926286    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:17.931855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:17.926365    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:18.938856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:18.938408    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:18.938856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:18.938485    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:19.944734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:19.944267    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 15:09:19.944734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:19.944334    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 15:09:21.168975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.168932    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:09:21.266031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.265994    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.298728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.298534    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:09:21.303978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.303943    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5gf\" (UniqueName: \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.304229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.304204    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.304428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.304412    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.304559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.304545    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.306121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.306094    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-100b2e-serving-cert\""
Oct 30 15:09:21.356936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.356896    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:09:21.358709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.358686    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:09:21.360547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.360498    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:09:21.405987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.405948    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.406167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.406056    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.406167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.406097    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.406325 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:09:21.406292    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-serving-cert: secret "maas-api-e2e-shared-b-w5-100b2e-serving-cert" not found
Oct 30 15:09:21.406474 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:09:21.406376    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls podName:c94160f9-510f-4ff4-a8b5-a8be745da01d nodeName:}" failed. No retries permitted until 2026-10-30 15:09:21.906354694 +0000 UTC m=+1311.348245409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls") pod "maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" (UID: "c94160f9-510f-4ff4-a8b5-a8be745da01d") : secret "maas-api-e2e-shared-b-w5-100b2e-serving-cert" not found
Oct 30 15:09:21.406573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.406551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5gf\" (UniqueName: \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.406953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.406927    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.409964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.409940    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.432553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.432482    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5gf\" (UniqueName: \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.929754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.929708    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.932328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.932301    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:21.938263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:21.938235    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:22.302977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:22.299110    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:09:22.304180 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:09:22.304109    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94160f9_510f_4ff4_a8b5_a8be745da01d.slice/crio-ed6649d0e6f568c89cccec8a6747a0fe02da87c4e0adc8cd4817a3d7c14b14c9 WatchSource:0}: Error finding container ed6649d0e6f568c89cccec8a6747a0fe02da87c4e0adc8cd4817a3d7c14b14c9: Status 404 returned error can't find the container with id ed6649d0e6f568c89cccec8a6747a0fe02da87c4e0adc8cd4817a3d7c14b14c9
Oct 30 15:09:23.016999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:23.016444    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" event={"ID":"c94160f9-510f-4ff4-a8b5-a8be745da01d","Type":"ContainerStarted","Data":"ed6649d0e6f568c89cccec8a6747a0fe02da87c4e0adc8cd4817a3d7c14b14c9"}
Oct 30 15:09:24.219892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:24.219154    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" event={"ID":"c94160f9-510f-4ff4-a8b5-a8be745da01d","Type":"ContainerStarted","Data":"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"}
Oct 30 15:09:24.220710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:24.220682    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:24.303295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:24.303228    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" podStartSLOduration=2.693778935 podStartE2EDuration="3.303204212s" totalImagesPullingTime="609.425277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:24.289674491 +0000 UTC m=+1313.731565210" watchObservedRunningTime="2026-10-30 15:09:24.303204212 +0000 UTC m=+1313.745094930"
Oct 30 15:09:27.745548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:27.745502    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:09:27.919863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:27.898129    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.006905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:27.971038    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f7acff45\"/\"openshift-service-ca.crt\""
Oct 30 15:09:28.037231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.013800    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f7acff45\"/\"default-dockercfg-9slwl\""
Oct 30 15:09:28.037231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.013910    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f7acff45\"/\"e2e-lifecycle-model-f7acff45-kserve-self-signed-certs\""
Oct 30 15:09:28.037231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.013976    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f7acff45\"/\"kube-root-ca.crt\""
Oct 30 15:09:28.104803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.099792    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111193    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111251    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111304    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111558    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111701    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.111791    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.221720    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.221872    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.221916    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.221971    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.222234    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.222467    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.226548    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.227232    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.228096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.227817    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.264339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.264263    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.270854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.266499    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.283861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.278149    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\") pod \"e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") " pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.473866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.472273    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:28.934944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:28.934911    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:09:28.935323 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:09:28.935211    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97927525_fb17_4e9c_80bd_8d2c47d0dc04.slice/crio-8a7847cea0a4e23da69605499e587ef5aa3fc786f58b420c3fe8cec863de5b51 WatchSource:0}: Error finding container 8a7847cea0a4e23da69605499e587ef5aa3fc786f58b420c3fe8cec863de5b51: Status 404 returned error can't find the container with id 8a7847cea0a4e23da69605499e587ef5aa3fc786f58b420c3fe8cec863de5b51
Oct 30 15:09:29.451854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:29.448305    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" event={"ID":"97927525-fb17-4e9c-80bd-8d2c47d0dc04","Type":"ContainerStarted","Data":"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"}
Oct 30 15:09:29.451854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:29.448362    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" event={"ID":"97927525-fb17-4e9c-80bd-8d2c47d0dc04","Type":"ContainerStarted","Data":"8a7847cea0a4e23da69605499e587ef5aa3fc786f58b420c3fe8cec863de5b51"}
Oct 30 15:09:29.474401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:29.473752    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:29.507853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:29.506212    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" podStartSLOduration=2.506188159 podStartE2EDuration="2.506188159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:29.49941706 +0000 UTC m=+1318.941307778" watchObservedRunningTime="2026-10-30 15:09:29.506188159 +0000 UTC m=+1318.948078877"
Oct 30 15:09:30.015854 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:09:30.015079    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:09:31.365395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:31.365364    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:09:33.367026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:33.365458    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:09:33.386864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:33.386071    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:09:38.473750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:38.473704    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:38.488142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:38.488108    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:39.091497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:39.091469    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:09:42.187224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.187131    5064 generic.go:356] "Generic (PLEG): container finished" podID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerID="8917048906aff05ef01bea6321a91643342a12ec82f5b5031cce20dd00695db2" exitCode=0
Oct 30 15:09:42.187224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.187198    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerDied","Data":"8917048906aff05ef01bea6321a91643342a12ec82f5b5031cce20dd00695db2"}
Oct 30 15:09:42.187717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.187702    5064 scope.go:122] "RemoveContainer" containerID="8917048906aff05ef01bea6321a91643342a12ec82f5b5031cce20dd00695db2"
Oct 30 15:09:42.190107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.190064    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:42.190235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.190174    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:42.190235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:42.190225    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:43.196266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:43.196229    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerStarted","Data":"7962a934896575b5aca27d4b6744e4287cf39a33e1c807aa94e078abd4b78630"}
Oct 30 15:09:43.204734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:43.204699    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:09:44.102337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:44.102298    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:09:44.103937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:44.103911    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 15:09:44.104063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:44.103974    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 15:09:45.102649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:45.102615    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 15:09:45.103127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:45.102676    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 15:09:46.102803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:46.102765    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 15:09:46.103254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:46.102867    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 15:09:47.102840 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:47.102790    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 15:09:47.103267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:47.102889    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 15:09:48.103259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:48.103220    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 15:09:48.104387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:48.103980    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 15:09:50.126701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:50.126657    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:09:50.127338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:50.127298    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:09:50.147138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:50.147108    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:09:50.167358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:50.167321    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:09:51.186960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.184011    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:09:51.195187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.195143    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:09:51.195370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.195356    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207354 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207323    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207424    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207590    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207638    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207691    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207765    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207935    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.207993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.207966    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2xj\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.208050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.208036    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.314888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.306915    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-76473c-openshift-default-dockercfg-p7vs4\""
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.318731    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.318780    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2xj\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.318880    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.318942    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.319033    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.319209    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.319247    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.319301    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.319376    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.320340    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.320619    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.335856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.322524    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.357653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.338227    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.379849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.370881    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.493143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.471467    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.750949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.709204    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:51.752694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:51.752658    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:52.066942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.063588    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2xj\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:52.133211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.133168    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:52.574956 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:09:52.571604    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ccde36_91c4_4d10_99d1_0e7b185bfe21.slice/crio-1de2f2468e1f860c0c74b8bb20b0eda73b56645d2e5bf2b817622da6d488e15d WatchSource:0}: Error finding container 1de2f2468e1f860c0c74b8bb20b0eda73b56645d2e5bf2b817622da6d488e15d: Status 404 returned error can't find the container with id 1de2f2468e1f860c0c74b8bb20b0eda73b56645d2e5bf2b817622da6d488e15d
Oct 30 15:09:52.581015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.579988    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:52.581015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.580185    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:52.581015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.580331    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:09:52.584992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.583541    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:09:52.676060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:52.676017    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerStarted","Data":"1de2f2468e1f860c0c74b8bb20b0eda73b56645d2e5bf2b817622da6d488e15d"}
Oct 30 15:09:53.730852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:53.725362    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerStarted","Data":"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"}
Oct 30 15:09:53.817750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:53.817680    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podStartSLOduration=2.817620842 podStartE2EDuration="2.817620842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:53.797013605 +0000 UTC m=+1343.238904324" watchObservedRunningTime="2026-10-30 15:09:53.817620842 +0000 UTC m=+1343.259511560"
Oct 30 15:09:54.139903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:54.139858    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:54.164790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:54.164752    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:54.775304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:54.775272    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:54.775824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:09:54.775344    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:09:59.823263 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:09:59.823156    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:10:02.339081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:02.339026    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:10:02.339529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:02.339103    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:10:04.035534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.035495    5064 generic.go:356] "Generic (PLEG): container finished" podID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerID="f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd" exitCode=0
Oct 30 15:10:04.036208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.035557    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerDied","Data":"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"}
Oct 30 15:10:04.036611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.036589    5064 scope.go:122] "RemoveContainer" containerID="f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"
Oct 30 15:10:04.038300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038280    5064 generic.go:356] "Generic (PLEG): container finished" podID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerID="2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148" exitCode=0
Oct 30 15:10:04.038396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038319    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerDied","Data":"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"}
Oct 30 15:10:04.038396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038334    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:04.038468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038408    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:04.038468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038446    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:04.038688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.038673    5064 scope.go:122] "RemoveContainer" containerID="2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"
Oct 30 15:10:04.040217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.040185    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:04.040320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.040253    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:04.040320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:04.040284    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:10:05.085863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.082905    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerStarted","Data":"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"}
Oct 30 15:10:05.086755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.086722    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:10:05.090784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.090753    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerStarted","Data":"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"}
Oct 30 15:10:05.091119 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.091102    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.199937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.199048    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4204df-serving-cert\""
Oct 30 15:10:05.202327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.201782    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.202327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.202275    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.202327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.202321    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.203283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.202526    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdrp\" (UniqueName: \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.209424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.209393    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:10:05.311160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.311126    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.311334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.311301    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.311334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.311325    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.311480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.311379    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdrp\" (UniqueName: \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.312691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.312368    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.377967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.361905    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:10:05.395945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.395376    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:05.395945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.395454    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:05.396197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.396000    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.499856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.498028    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdrp\" (UniqueName: \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.500237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.500207    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\") pod \"maas-api-e2e-api-a-4204df-f96498fbd-bnskz\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.505291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.505225    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:05.521852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.516898    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:10:05.521852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.517234    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:05.521852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.517298    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:05.624850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:05.621262    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:10:06.395908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:06.388800    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:06.395908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:06.388882    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:06.548864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:06.548341    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:06.548864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:06.548406    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:07.492853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:07.492772    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:07.493380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:07.492853    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:07.514403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:07.513556    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:07.514403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:07.513638    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:07.696539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:07.696502    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:10:08.333553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:08.333514    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:08.334285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:08.334251    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:08.533868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:08.533814    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:08.534493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:08.534454    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:08.609692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:08.609654    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" event={"ID":"630c7525-1a95-4288-bf16-52750f26e351","Type":"ContainerStarted","Data":"5cee3e0c15513a360f5dfea56a58278ade7f72eb4b8e66ea2759bc42dfcf1a3b"}
Oct 30 15:10:09.346853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.346448    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:09.346853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.346519    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:09.534854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.522204    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:09.534854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.522270    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:09.653857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.648823    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" event={"ID":"630c7525-1a95-4288-bf16-52750f26e351","Type":"ContainerStarted","Data":"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"}
Oct 30 15:10:09.653857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.650165    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:09.703819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:09.703764    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" podStartSLOduration=4.217859268 podStartE2EDuration="4.703747833s" totalImagesPullingTime="485.888565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:09.700281646 +0000 UTC m=+1359.142172362" watchObservedRunningTime="2026-10-30 15:10:09.703747833 +0000 UTC m=+1359.145638616"
Oct 30 15:10:10.331854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:10.331265    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:10.331854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:10.331333    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:10.535859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:10.517250    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:10.535859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:10.517315    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:11.340855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:11.335686    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 15:10:11.340855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:11.335751    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 15:10:11.519415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:11.519374    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 15:10:11.519730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:11.519681    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 15:10:12.367304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:12.367262    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:10:12.390207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:12.390170    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:10:12.520188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:12.520158    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:10:12.521026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:12.520998    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:10:12.522151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:12.522127    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:10:16.726734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:16.726695    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:10:29.631701 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:10:29.631658    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:10:31.458057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.458019    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:10:31.462449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.462427    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.483566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.483535    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:10:31.503208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.503173    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-76473c-serving-cert\""
Oct 30 15:10:31.556455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.556415    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.556670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.556578    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.556760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.556681    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.556990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.556968    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpgh\" (UniqueName: \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.678867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.674287    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.678867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.674430    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.678867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.674523    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.678867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.674804    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpgh\" (UniqueName: \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.678867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.676212    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.719441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.719363    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.720330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.720305    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.726517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.726437    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpgh\" (UniqueName: \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\") pod \"maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:31.780863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:31.779813    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:32.175581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:32.175547    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:10:32.175870 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:10:32.175808    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89094cc8_092a_4281_ace4_52ef6af5718c.slice/crio-3ef9d582b6d6bb5f0cd8a49d0c30889611297bdc71bc5902360c106ecb235f0c WatchSource:0}: Error finding container 3ef9d582b6d6bb5f0cd8a49d0c30889611297bdc71bc5902360c106ecb235f0c: Status 404 returned error can't find the container with id 3ef9d582b6d6bb5f0cd8a49d0c30889611297bdc71bc5902360c106ecb235f0c
Oct 30 15:10:32.177601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:32.177579    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:10:33.047857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:33.044266    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" event={"ID":"89094cc8-092a-4281-ace4-52ef6af5718c","Type":"ContainerStarted","Data":"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"}
Oct 30 15:10:33.047857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:33.044346    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" event={"ID":"89094cc8-092a-4281-ace4-52ef6af5718c","Type":"ContainerStarted","Data":"3ef9d582b6d6bb5f0cd8a49d0c30889611297bdc71bc5902360c106ecb235f0c"}
Oct 30 15:10:33.064950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:33.057253    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:33.111995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:33.111930    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" podStartSLOduration=1.688105822 podStartE2EDuration="2.111818715s" totalImagesPullingTime="423.712893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:33.105776048 +0000 UTC m=+1382.547666778" watchObservedRunningTime="2026-10-30 15:10:33.111818715 +0000 UTC m=+1382.553709432"
Oct 30 15:10:40.111229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:40.111196    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:10:45.236977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:45.236940    5064 generic.go:356] "Generic (PLEG): container finished" podID="a41b224f-88bf-4956-a1dd-505bdb9a9e54" containerID="6775333de6cb4f9b6b6a0eca8cf8d0efcb8a02d54b6a630ce3aca1da85634cc2" exitCode=0
Oct 30 15:10:45.237394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:45.237017    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v" event={"ID":"a41b224f-88bf-4956-a1dd-505bdb9a9e54","Type":"ContainerDied","Data":"6775333de6cb4f9b6b6a0eca8cf8d0efcb8a02d54b6a630ce3aca1da85634cc2"}
Oct 30 15:10:46.443176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.443146    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.507885    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\") pod \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") "
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.508025    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\") pod \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") "
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.508065    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\") pod \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") "
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.508100    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\" (UniqueName: \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\") pod \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\" (UID: \"a41b224f-88bf-4956-a1dd-505bdb9a9e54\") "
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.520459    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle" pod "a41b224f-88bf-4956-a1dd-505bdb9a9e54" (UID: "a41b224f-88bf-4956-a1dd-505bdb9a9e54"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.520850    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle" pod "a41b224f-88bf-4956-a1dd-505bdb9a9e54" (UID: "a41b224f-88bf-4956-a1dd-505bdb9a9e54"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:10:46.522063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.521335    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path" pod "a41b224f-88bf-4956-a1dd-505bdb9a9e54" (UID: "a41b224f-88bf-4956-a1dd-505bdb9a9e54"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:10:46.548856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.542066    5064 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a41b224f-88bf-4956-a1dd-505bdb9a9e54-archives-path\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:10:46.548856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.542102    5064 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-service-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:10:46.548856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.542122    5064 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a41b224f-88bf-4956-a1dd-505bdb9a9e54-trusted-ca-bundle\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:10:46.557566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.557193    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q" pod "a41b224f-88bf-4956-a1dd-505bdb9a9e54" (UID: "a41b224f-88bf-4956-a1dd-505bdb9a9e54"). InnerVolumeSpecName "kube-api-access-nr89q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:10:46.648853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:46.648561    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr89q\" (UniqueName: \"kubernetes.io/projected/a41b224f-88bf-4956-a1dd-505bdb9a9e54-kube-api-access-nr89q\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:10:47.268211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:47.268181    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v"
Oct 30 15:10:47.268414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:47.268213    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mhmsw-5fr7v" event={"ID":"a41b224f-88bf-4956-a1dd-505bdb9a9e54","Type":"ContainerDied","Data":"de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70"}
Oct 30 15:10:47.268414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:47.268274    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1a4ad13236a3718d131f190d373564a037367d2f81bc7f6d762ba85a791d70"
Oct 30 15:10:55.429625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:55.429589    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:10:55.430086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:10:55.429784    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy" containerID="cri-o://4b7ae868088a871c8cb235f6d243a338da96701778eaa1da3eeed2a75ff2e6a7" gracePeriod=30
Oct 30 15:10:59.879905 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:10:59.876517    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:11:01.766322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:01.763957    5064 generic.go:356] "Generic (PLEG): container finished" podID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerID="4b7ae868088a871c8cb235f6d243a338da96701778eaa1da3eeed2a75ff2e6a7" exitCode=0
Oct 30 15:11:01.766322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:01.764260    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" event={"ID":"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8","Type":"ContainerDied","Data":"4b7ae868088a871c8cb235f6d243a338da96701778eaa1da3eeed2a75ff2e6a7"}
Oct 30 15:11:01.975854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:01.967975    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:11:02.026869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.026748    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.027242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.027187    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.027424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.027407    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.027614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.027583    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.027756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.027741    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.028043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.028023    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.028201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.028183    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.028420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.028402    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.028622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.028605    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\") pod \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\" (UID: \"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8\") "
Oct 30 15:11:02.029480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.029392    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:02.117859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.055658    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:02.118363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.082121    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:02.118554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.082435    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:02.122903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.088366    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.090987    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.099389    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.099529    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "kube-api-access-4v2mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.111964    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket" pod "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" (UID: "38e0daa3-f81e-4b68-9c4c-ff683d9d1de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132721    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132752    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132768    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132782    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132795    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132808    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2mg\" (UniqueName: \"kubernetes.io/projected/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-kube-api-access-4v2mg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132822    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132857    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:02.145873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:02.132871    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:03.026353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:03.026318    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"
Oct 30 15:11:03.217853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:03.216395    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64" event={"ID":"38e0daa3-f81e-4b68-9c4c-ff683d9d1de8","Type":"ContainerDied","Data":"68a5d02ed392d6f7c16bf9bd19fec5ae4feb2e27dcde47c1155fa69bd3ec5b37"}
Oct 30 15:11:03.217853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:03.216451    5064 scope.go:122] "RemoveContainer" containerID="4b7ae868088a871c8cb235f6d243a338da96701778eaa1da3eeed2a75ff2e6a7"
Oct 30 15:11:03.235270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:03.235218    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:11:03.242239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:03.242184    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-vmc64"]
Oct 30 15:11:04.786852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:04.781087    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" path="/var/lib/kubelet/pods/38e0daa3-f81e-4b68-9c4c-ff683d9d1de8/volumes"
Oct 30 15:11:10.593367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.593324    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:11:10.593880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.593586    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy" containerID="cri-o://42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30" gracePeriod=30
Oct 30 15:11:10.599163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.599130    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:11:10.599416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.599389    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy" containerID="cri-o://f3d6b9f022282a201329619a26c877425e854e49e6ee4fd5ae39795e0640e55a" gracePeriod=30
Oct 30 15:11:10.604285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.604257    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:11:10.604485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.604465    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy" containerID="cri-o://a6ac4cbab3e014bc22d6fbe8828ef17a9a3e67657a4f8bada60294bc0f99dedf" gracePeriod=30
Oct 30 15:11:10.815023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.814980    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:11:10.815205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:10.815041    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:11:11.095320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:11.095284    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:11:11.095534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:11.095512    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy" containerID="cri-o://032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc" gracePeriod=30
Oct 30 15:11:12.123787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.123752    5064 generic.go:356] "Generic (PLEG): container finished" podID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerID="8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b" exitCode=0
Oct 30 15:11:12.124388 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.123873    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerDied","Data":"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"}
Oct 30 15:11:12.124519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.124500    5064 scope.go:122] "RemoveContainer" containerID="8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"
Oct 30 15:11:12.126593 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.126554    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:12.126680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.126632    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:12.126680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:12.126666    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:13.145084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:13.145044    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerStarted","Data":"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"}
Oct 30 15:11:14.137001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:14.134255    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:11:14.137001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:14.134411    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 15:11:14.137001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:14.134455    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 15:11:15.137858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:15.137256    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 15:11:15.137858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:15.137321    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 15:11:16.074559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.074530    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:11:16.156088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.155046    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 15:11:16.156088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.155098    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199071    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199130    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199313    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199365    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199422    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199485    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199597    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199641    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.199732    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\") pod \"425ae39f-9069-43e9-98f1-671f1eaafc5b\" (UID: \"425ae39f-9069-43e9-98f1-671f1eaafc5b\") "
Oct 30 15:11:16.202216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.201910    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.202992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.202239    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:16.202992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.202787    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.203363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.203273    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.204025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.203986    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.210912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.208942    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:16.210912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.209307    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.227857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.221360    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "kube-api-access-5lcj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.252075    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token" pod "425ae39f-9069-43e9-98f1-671f1eaafc5b" (UID: "425ae39f-9069-43e9-98f1-671f1eaafc5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.255256    5064 generic.go:356] "Generic (PLEG): container finished" podID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerID="42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30" exitCode=0
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.255416    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" event={"ID":"425ae39f-9069-43e9-98f1-671f1eaafc5b","Type":"ContainerDied","Data":"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"}
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.255476    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds" event={"ID":"425ae39f-9069-43e9-98f1-671f1eaafc5b","Type":"ContainerDied","Data":"86c03675f6fcc6e0d1ed1f7da3b5a05ab07b72819ccb4ac51184cb8ab43abf9c"}
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.255510    5064 scope.go:122] "RemoveContainer" containerID="42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"
Oct 30 15:11:16.256511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.255704    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"
Oct 30 15:11:16.279082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.279017    5064 generic.go:356] "Generic (PLEG): container finished" podID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerID="a6ac4cbab3e014bc22d6fbe8828ef17a9a3e67657a4f8bada60294bc0f99dedf" exitCode=0
Oct 30 15:11:16.279577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.279522    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" event={"ID":"f13f4b1c-8da8-4b6e-94cb-78beb8086558","Type":"ContainerDied","Data":"a6ac4cbab3e014bc22d6fbe8828ef17a9a3e67657a4f8bada60294bc0f99dedf"}
Oct 30 15:11:16.283277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.283252    5064 generic.go:356] "Generic (PLEG): container finished" podID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerID="f3d6b9f022282a201329619a26c877425e854e49e6ee4fd5ae39795e0640e55a" exitCode=0
Oct 30 15:11:16.284557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.284496    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" event={"ID":"03dfeb13-f0d5-4f47-b20d-c3a131443ae9","Type":"ContainerDied","Data":"f3d6b9f022282a201329619a26c877425e854e49e6ee4fd5ae39795e0640e55a"}
Oct 30 15:11:16.303028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.302998    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303030    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425ae39f-9069-43e9-98f1-671f1eaafc5b-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303044    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303058    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lcj7\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-kube-api-access-5lcj7\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303071    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303084    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303095    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303107    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.303276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.303118    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425ae39f-9069-43e9-98f1-671f1eaafc5b-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.307746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.307720    5064 scope.go:122] "RemoveContainer" containerID="42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"
Oct 30 15:11:16.308876 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:16.308817    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30\": container with ID starting with 42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30 not found: ID does not exist" containerID="42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"
Oct 30 15:11:16.308998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.308879    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30"} err="failed to get container status \"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30\": rpc error: code = NotFound desc = could not find container \"42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30\": container with ID starting with 42af548bc5c7fff114fb7517d5b1d7a6b8d63ebcf88a6de601a1d46a24896a30 not found: ID does not exist"
Oct 30 15:11:16.342870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.340500    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:11:16.374947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.374810    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:11:16.386987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.386953    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:11:16.398383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.398344    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-9h9ds"]
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404172    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404229    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404258    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404297    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404334    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.404427    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.405646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.405444    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.406384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.405899    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.406519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.406487    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.407980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.407956    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.431861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.422156    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.466006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.465963    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.505785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.505741    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.506003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.505802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.505874    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.505931    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506199    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506260    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.506343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506289    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.506343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506324    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506398    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506494    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.506605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506528    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\") pod \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\" (UID: \"f13f4b1c-8da8-4b6e-94cb-78beb8086558\") "
Oct 30 15:11:16.506668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.506641    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\") pod \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\" (UID: \"03dfeb13-f0d5-4f47-b20d-c3a131443ae9\") "
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507172    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507690    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507719    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507734    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507766    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507782    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507795    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.507877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.507808    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.508569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.508380    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:16.559014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.509601    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.559014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.510415    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.559014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.511086    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "kube-api-access-gx6dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.559014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.511103    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.559014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.541009    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:16.560011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.559895    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.563267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.563230    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "kube-api-access-48xh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.563433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.563392    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:16.568084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.568047    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token" pod "f13f4b1c-8da8-4b6e-94cb-78beb8086558" (UID: "f13f4b1c-8da8-4b6e-94cb-78beb8086558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.570437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.570405    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo" pod "03dfeb13-f0d5-4f47-b20d-c3a131443ae9" (UID: "03dfeb13-f0d5-4f47-b20d-c3a131443ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:16.576549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.576522    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:11:16.608888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.608855    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.608941    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609104    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609175    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.609246 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609204    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609293    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609328    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609381    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.609402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609393    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609460    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609510 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609492    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.609588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609542    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\") pod \"032a0871-0784-4069-9be1-8bb563921baf\" (UID: \"032a0871-0784-4069-9be1-8bb563921baf\") "
Oct 30 15:11:16.609588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.609570    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.610042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610016    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:16.610434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610408    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610439    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610454    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13f4b1c-8da8-4b6e-94cb-78beb8086558-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610469    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610484    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610497    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610509    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610523    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610537    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13f4b1c-8da8-4b6e-94cb-78beb8086558-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610548    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610561    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610575    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032a0871-0784-4069-9be1-8bb563921baf-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610587    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610601    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6dn\" (UniqueName: \"kubernetes.io/projected/f13f4b1c-8da8-4b6e-94cb-78beb8086558-kube-api-access-gx6dn\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610615    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48xh8\" (UniqueName: \"kubernetes.io/projected/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-kube-api-access-48xh8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.610922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.610629    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dfeb13-f0d5-4f47-b20d-c3a131443ae9-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.611752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.611729    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:16.611930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.611761    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:16.611930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.611891    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.612039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.611945    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4" pod "032a0871-0784-4069-9be1-8bb563921baf" (UID: "032a0871-0784-4069-9be1-8bb563921baf"). InnerVolumeSpecName "kube-api-access-dvkq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:16.712816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.712779    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.712816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.712817    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032a0871-0784-4069-9be1-8bb563921baf-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.712816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.712854    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkq4\" (UniqueName: \"kubernetes.io/projected/032a0871-0784-4069-9be1-8bb563921baf-kube-api-access-dvkq4\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.713146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.712873    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032a0871-0784-4069-9be1-8bb563921baf-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:16.769995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:16.766721    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" path="/var/lib/kubelet/pods/425ae39f-9069-43e9-98f1-671f1eaafc5b/volumes"
Oct 30 15:11:17.174220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.162888    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 15:11:17.174220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.162953    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 15:11:17.295575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.295537    5064 generic.go:356] "Generic (PLEG): container finished" podID="032a0871-0784-4069-9be1-8bb563921baf" containerID="032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc" exitCode=0
Oct 30 15:11:17.295772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.295598    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"
Oct 30 15:11:17.295772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.295627    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" event={"ID":"032a0871-0784-4069-9be1-8bb563921baf","Type":"ContainerDied","Data":"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"}
Oct 30 15:11:17.295772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.295706    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx" event={"ID":"032a0871-0784-4069-9be1-8bb563921baf","Type":"ContainerDied","Data":"fa41a7553bf1a8788b956079da8cf4906c0cc52889f075c8bfbd196b3e7f8d3b"}
Oct 30 15:11:17.295772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.295739    5064 scope.go:122] "RemoveContainer" containerID="032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"
Oct 30 15:11:17.297458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.297433    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn" event={"ID":"f13f4b1c-8da8-4b6e-94cb-78beb8086558","Type":"ContainerDied","Data":"9e85e9a85083ab785f50e65970751eac876ab8a9bfc476f47f9ed1dc777c3011"}
Oct 30 15:11:17.297747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.297531    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"
Oct 30 15:11:17.300597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.300575    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"
Oct 30 15:11:17.300786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.300577    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg" event={"ID":"03dfeb13-f0d5-4f47-b20d-c3a131443ae9","Type":"ContainerDied","Data":"5cc8c5dff9348c4457686f4df7bd5a4aaefd24d13beb067024ed0c64fb772913"}
Oct 30 15:11:17.309537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.309518    5064 scope.go:122] "RemoveContainer" containerID="032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"
Oct 30 15:11:17.309858 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:17.309823    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc\": container with ID starting with 032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc not found: ID does not exist" containerID="032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"
Oct 30 15:11:17.309956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.309866    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc"} err="failed to get container status \"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc\": rpc error: code = NotFound desc = could not find container \"032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc\": container with ID starting with 032c2b2fc3ce2a9b13fd89db1acc7fd05a136d4ea7c1adea388a971757eeb7bc not found: ID does not exist"
Oct 30 15:11:17.309956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.309885    5064 scope.go:122] "RemoveContainer" containerID="a6ac4cbab3e014bc22d6fbe8828ef17a9a3e67657a4f8bada60294bc0f99dedf"
Oct 30 15:11:17.350571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.350539    5064 scope.go:122] "RemoveContainer" containerID="f3d6b9f022282a201329619a26c877425e854e49e6ee4fd5ae39795e0640e55a"
Oct 30 15:11:17.366903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.366874    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:11:17.380599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.380571    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-vknvn"]
Oct 30 15:11:17.433909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.431721    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:11:17.446345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.446315    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qxvjx"]
Oct 30 15:11:17.486227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.486195    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:11:17.492283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:17.492254    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-4wpsg"]
Oct 30 15:11:18.138017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:18.137974    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 15:11:18.138220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:18.138041    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 15:11:18.728403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:18.728370    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032a0871-0784-4069-9be1-8bb563921baf" path="/var/lib/kubelet/pods/032a0871-0784-4069-9be1-8bb563921baf/volumes"
Oct 30 15:11:18.729088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:18.729067    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" path="/var/lib/kubelet/pods/03dfeb13-f0d5-4f47-b20d-c3a131443ae9/volumes"
Oct 30 15:11:18.729628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:18.729613    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" path="/var/lib/kubelet/pods/f13f4b1c-8da8-4b6e-94cb-78beb8086558/volumes"
Oct 30 15:11:19.264520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.264484    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:11:19.265963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.265916    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:11:19.267028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.267002    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:11:19.811020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.810987    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:11:19.812014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.811991    5064 cpu_manager.go:414] "containerMap: removing container" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy"
Oct 30 15:11:19.812014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812012    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812037    5064 cpu_manager.go:414] "containerMap: removing container" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812045    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812114    5064 cpu_manager.go:414] "containerMap: removing container" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812122    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812141    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a41b224f-88bf-4956-a1dd-505bdb9a9e54" containerName="insights-gathering"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812148    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41b224f-88bf-4956-a1dd-505bdb9a9e54" containerName="insights-gathering"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812159    5064 cpu_manager.go:414] "containerMap: removing container" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812167    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812177    5064 cpu_manager.go:414] "containerMap: removing container" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy"
Oct 30 15:11:19.812212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812185    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812349    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41b224f-88bf-4956-a1dd-505bdb9a9e54" containerName="insights-gathering"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812366    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e0daa3-f81e-4b68-9c4c-ff683d9d1de8" containerName="istio-proxy"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812378    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="032a0871-0784-4069-9be1-8bb563921baf" containerName="istio-proxy"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812392    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="425ae39f-9069-43e9-98f1-671f1eaafc5b" containerName="istio-proxy"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812404    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13f4b1c-8da8-4b6e-94cb-78beb8086558" containerName="istio-proxy"
Oct 30 15:11:19.812668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.812424    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="03dfeb13-f0d5-4f47-b20d-c3a131443ae9" containerName="istio-proxy"
Oct 30 15:11:19.816663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.816639    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.847484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.847447    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:11:19.847786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.847764    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-147bdb-openshift-default-dockercfg-q7tjn\""
Oct 30 15:11:19.898227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898189    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898301    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhzf\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898430    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898489    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898698    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898763    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898893    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:19.898976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:19.898960    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000398    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhzf\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000508    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000630    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000685    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000727    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000820    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.000925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000882    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.001039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.000994    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.002350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.001315    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.002350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.001346    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.002350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.001326    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.002350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.001428    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.002350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.001464    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.004870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.003284    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.005486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.005463    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.056545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.056509    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.061643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.061556    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhzf\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\") pod \"e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") " pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.137853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.137553    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:20.747565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.747536    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:11:20.747969 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:20.747945    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00805fd_dea5_414c_8a93_4a7af0dc70df.slice/crio-f07c819a5331df2ad7965870bc511a856dddcbda598876576135bc67abcde71f WatchSource:0}: Error finding container f07c819a5331df2ad7965870bc511a856dddcbda598876576135bc67abcde71f: Status 404 returned error can't find the container with id f07c819a5331df2ad7965870bc511a856dddcbda598876576135bc67abcde71f
Oct 30 15:11:20.750771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.750713    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:20.750837 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.750781    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:20.750837 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:20.750811    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:21.525394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:21.525356    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerStarted","Data":"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"}
Oct 30 15:11:21.525850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:21.525411    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerStarted","Data":"f07c819a5331df2ad7965870bc511a856dddcbda598876576135bc67abcde71f"}
Oct 30 15:11:21.591878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:21.591807    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podStartSLOduration=2.591792431 podStartE2EDuration="2.591792431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:21.588216765 +0000 UTC m=+1431.030107483" watchObservedRunningTime="2026-10-30 15:11:21.591792431 +0000 UTC m=+1431.033683148"
Oct 30 15:11:22.141537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:22.141498    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:22.147867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:22.147821    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:22.542973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:22.542367    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:22.599893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:22.594169    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:11:26.127496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.127448    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:11:26.128378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.127814    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy" containerID="cri-o://e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b" gracePeriod=30
Oct 30 15:11:26.323020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.322988    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9025-ad92b5"]
Oct 30 15:11:26.329416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.329391    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:26.344489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.342756    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-ad92b5"]
Oct 30 15:11:26.422906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.422688    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\""
Oct 30 15:11:26.567853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.489039    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwg8\" (UniqueName: \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\") pod \"test-curl-9025-ad92b5\" (UID: \"59de83e0-71d3-4add-aeec-d20649177457\") " pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:26.775902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.771798    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwg8\" (UniqueName: \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\") pod \"test-curl-9025-ad92b5\" (UID: \"59de83e0-71d3-4add-aeec-d20649177457\") " pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:26.902497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.902458    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwg8\" (UniqueName: \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\") pod \"test-curl-9025-ad92b5\" (UID: \"59de83e0-71d3-4add-aeec-d20649177457\") " pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:26.977931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:26.977892    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:27.285413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:27.285338    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-ad92b5"]
Oct 30 15:11:27.285879 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:27.285574    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59de83e0_71d3_4add_aeec_d20649177457.slice/crio-4e88b6f997c93953154372922de016ffa44a83fbcf7f55bf1fb8a792b7c77901 WatchSource:0}: Error finding container 4e88b6f997c93953154372922de016ffa44a83fbcf7f55bf1fb8a792b7c77901: Status 404 returned error can't find the container with id 4e88b6f997c93953154372922de016ffa44a83fbcf7f55bf1fb8a792b7c77901
Oct 30 15:11:27.837623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:27.837581    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-ad92b5" event={"ID":"59de83e0-71d3-4add-aeec-d20649177457","Type":"ContainerStarted","Data":"ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0"}
Oct 30 15:11:27.837904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:27.837635    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-ad92b5" event={"ID":"59de83e0-71d3-4add-aeec-d20649177457","Type":"ContainerStarted","Data":"4e88b6f997c93953154372922de016ffa44a83fbcf7f55bf1fb8a792b7c77901"}
Oct 30 15:11:27.882869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:27.882799    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9025-ad92b5" podStartSLOduration=1.8827844059999999 podStartE2EDuration="1.882784406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:27.878803039 +0000 UTC m=+1437.320693755" watchObservedRunningTime="2026-10-30 15:11:27.882784406 +0000 UTC m=+1437.324675117"
Oct 30 15:11:28.856531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:28.856496    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9025-ad92b5"]
Oct 30 15:11:28.862069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:28.862037    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9025-ad92b5"]
Oct 30 15:11:29.011502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.011461    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9025-c185ce"]
Oct 30 15:11:29.012272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.012256    5064 cpu_manager.go:414] "containerMap: removing container" podUID="59de83e0-71d3-4add-aeec-d20649177457" containerName="test-curl-9025-ad92b5"
Oct 30 15:11:29.012326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.012272    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59de83e0-71d3-4add-aeec-d20649177457" containerName="test-curl-9025-ad92b5"
Oct 30 15:11:29.012473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.012461    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="59de83e0-71d3-4add-aeec-d20649177457" containerName="test-curl-9025-ad92b5"
Oct 30 15:11:29.017041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.017023    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:29.032736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.032706    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-c185ce"]
Oct 30 15:11:29.058346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.058302    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55s8w\" (UniqueName: \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\") pod \"test-curl-9025-c185ce\" (UID: \"7ca71dda-80f4-4e81-ba74-02807fec0331\") " pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:29.187852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.181321    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55s8w\" (UniqueName: \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\") pod \"test-curl-9025-c185ce\" (UID: \"7ca71dda-80f4-4e81-ba74-02807fec0331\") " pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:29.212513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.211885    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55s8w\" (UniqueName: \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\") pod \"test-curl-9025-c185ce\" (UID: \"7ca71dda-80f4-4e81-ba74-02807fec0331\") " pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:29.336122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.334743    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:29.719205 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:29.718609    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:11:29.854323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.854251    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9025-ad92b5" podUID="59de83e0-71d3-4add-aeec-d20649177457" containerName="test-curl-9025-ad92b5" containerID="cri-o://ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0" gracePeriod=2
Oct 30 15:11:29.869764 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:29.869722    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-ad92b5\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="59de83e0-71d3-4add-aeec-d20649177457" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:29.996711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:29.996672    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-c185ce"]
Oct 30 15:11:29.997667 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:29.997635    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ca71dda_80f4_4e81_ba74_02807fec0331.slice/crio-08634057736b4b3ffb278650323688398d7904207e147b5ed77af5659897e71b WatchSource:0}: Error finding container 08634057736b4b3ffb278650323688398d7904207e147b5ed77af5659897e71b: Status 404 returned error can't find the container with id 08634057736b4b3ffb278650323688398d7904207e147b5ed77af5659897e71b
Oct 30 15:11:30.735874 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:30.735811    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-ad92b5\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="59de83e0-71d3-4add-aeec-d20649177457" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:30.864042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:30.864005    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-c185ce" event={"ID":"7ca71dda-80f4-4e81-ba74-02807fec0331","Type":"ContainerStarted","Data":"ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6"}
Oct 30 15:11:30.864042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:30.864057    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-c185ce" event={"ID":"7ca71dda-80f4-4e81-ba74-02807fec0331","Type":"ContainerStarted","Data":"08634057736b4b3ffb278650323688398d7904207e147b5ed77af5659897e71b"}
Oct 30 15:11:30.907801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:30.907746    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9025-c185ce" podStartSLOduration=2.907730665 podStartE2EDuration="2.907730665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:30.905185614 +0000 UTC m=+1440.347076333" watchObservedRunningTime="2026-10-30 15:11:30.907730665 +0000 UTC m=+1440.349621382"
Oct 30 15:11:31.519867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.509658    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.559698    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.559886    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560043    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560164    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560235    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560306    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560452    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560767    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.560823    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\") pod \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\" (UID: \"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8\") "
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.561171    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.561434    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.561673    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:31.562045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.561905    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:31.563080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.562302    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:11:31.568176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568139    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "kube-api-access-dc8mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:31.568340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568199    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.568340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568228    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.568340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568242    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.568340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568274    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.568340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.568294    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.581637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.573484    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:31.581637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.573653    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:11:31.581637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.574956    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo" pod "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" (UID: "2a14ea05-11b3-4e04-ae5e-e9e01648c8c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:11:31.682915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.674298    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.682915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.674340    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8mg\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-kube-api-access-dc8mg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.682915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.674358    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.682915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.674376    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:31.875045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.875016    5064 generic.go:356] "Generic (PLEG): container finished" podID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerID="e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b" exitCode=0
Oct 30 15:11:31.875129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.875076    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"
Oct 30 15:11:31.875129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.875102    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" event={"ID":"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8","Type":"ContainerDied","Data":"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"}
Oct 30 15:11:31.875213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.875161    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx" event={"ID":"2a14ea05-11b3-4e04-ae5e-e9e01648c8c8","Type":"ContainerDied","Data":"7481f96333b496945072ed784b26a30c5ffab20b35467d9b4e0db5f303bdfeb2"}
Oct 30 15:11:31.875213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.875188    5064 scope.go:122] "RemoveContainer" containerID="e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"
Oct 30 15:11:31.923801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.923775    5064 scope.go:122] "RemoveContainer" containerID="e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"
Oct 30 15:11:31.924200 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:31.924180    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b\": container with ID starting with e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b not found: ID does not exist" containerID="e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"
Oct 30 15:11:31.924240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.924215    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b"} err="failed to get container status \"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b\": rpc error: code = NotFound desc = could not find container \"e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b\": container with ID starting with e3c6264250900a77a821ed216a47bd65209944ee55c9c9a1b83c5d01f1293c4b not found: ID does not exist"
Oct 30 15:11:31.963899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.963678    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:11:31.976945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:31.976905    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-cplkx"]
Oct 30 15:11:32.027499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.027397    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9025-c185ce"]
Oct 30 15:11:32.034344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.034295    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9025-c185ce"]
Oct 30 15:11:32.089711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.088191    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:32.186195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.186154    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\" (UniqueName: \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\") pod \"59de83e0-71d3-4add-aeec-d20649177457\" (UID: \"59de83e0-71d3-4add-aeec-d20649177457\") "
Oct 30 15:11:32.189922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.189882    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8" pod "59de83e0-71d3-4add-aeec-d20649177457" (UID: "59de83e0-71d3-4add-aeec-d20649177457"). InnerVolumeSpecName "kube-api-access-vjwg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:32.220114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.220020    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9025-0e87a3"]
Oct 30 15:11:32.220873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.220821    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy"
Oct 30 15:11:32.220873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.220854    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy"
Oct 30 15:11:32.221013 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.220890    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" containerName="test-curl-9025-c185ce"
Oct 30 15:11:32.221013 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.220896    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" containerName="test-curl-9025-c185ce"
Oct 30 15:11:32.221084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.221028    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" containerName="istio-proxy"
Oct 30 15:11:32.221084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.221041    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" containerName="test-curl-9025-c185ce"
Oct 30 15:11:32.225166 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.225138    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:32.243163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.243127    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-0e87a3"]
Oct 30 15:11:32.289215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.289178    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5s5\" (UniqueName: \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\") pod \"test-curl-9025-0e87a3\" (UID: \"1420f5fe-20f0-4cce-9e5f-9ee55a326f80\") " pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:32.289592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.289573    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwg8\" (UniqueName: \"kubernetes.io/projected/59de83e0-71d3-4add-aeec-d20649177457-kube-api-access-vjwg8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:32.390923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.390877    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5s5\" (UniqueName: \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\") pod \"test-curl-9025-0e87a3\" (UID: \"1420f5fe-20f0-4cce-9e5f-9ee55a326f80\") " pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:32.416867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.416805    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5s5\" (UniqueName: \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\") pod \"test-curl-9025-0e87a3\" (UID: \"1420f5fe-20f0-4cce-9e5f-9ee55a326f80\") " pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:32.437558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.437521    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:11:32.448219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.448188    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.469889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.469805    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:11:32.503000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.502969    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24fff1-openshift-default-dockercfg-2nj9t\""
Oct 30 15:11:32.549985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.549949    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:32.595049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595007    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595051    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595182    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595235    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595432    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595467    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmmq\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595498    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595542    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.595655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.595625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.713647    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.713878    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.713916    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714027    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714092    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714255    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714287    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmmq\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714317    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.734906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.714340    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.768188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.745879    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.768188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.746171    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.768188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.746814    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.768188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.747437    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:32.768188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:32.747997    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.050480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.044999    5064 generic.go:356] "Generic (PLEG): container finished" podID="59de83e0-71d3-4add-aeec-d20649177457" containerID="ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0" exitCode=137
Oct 30 15:11:33.050480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.045191    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:33.082896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.082865    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.100617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.098163    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.104473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.103108    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.106083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.105994    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9025-c185ce" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" containerName="test-curl-9025-c185ce" containerID="cri-o://ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6" gracePeriod=2
Oct 30 15:11:33.119916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.112369    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmmq\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.119916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.118117    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a14ea05-11b3-4e04-ae5e-e9e01648c8c8" path="/var/lib/kubelet/pods/2a14ea05-11b3-4e04-ae5e-e9e01648c8c8/volumes"
Oct 30 15:11:33.119916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.118882    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59de83e0-71d3-4add-aeec-d20649177457" path="/var/lib/kubelet/pods/59de83e0-71d3-4add-aeec-d20649177457/volumes"
Oct 30 15:11:33.119916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.119909    5064 scope.go:122] "RemoveContainer" containerID="ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0"
Oct 30 15:11:33.134916 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:33.133799    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-ad92b5\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="59de83e0-71d3-4add-aeec-d20649177457" pod="opendatahub/test-curl-9025-ad92b5"
Oct 30 15:11:33.152758 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.152737    5064 scope.go:122] "RemoveContainer" containerID="ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0"
Oct 30 15:11:33.153159 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:33.153135    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0\": container with ID starting with ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0 not found: ID does not exist" containerID="ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0"
Oct 30 15:11:33.153243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.153167    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0"} err="failed to get container status \"ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0\": rpc error: code = NotFound desc = could not find container \"ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0\": container with ID starting with ebabc7cb75e089e48789d682dd4d8ce2b14040b38e3db371391629735f153ae0 not found: ID does not exist"
Oct 30 15:11:33.265132 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:33.265093    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1420f5fe_20f0_4cce_9e5f_9ee55a326f80.slice/crio-50eb532f2d3f38a8a78d822ae6e3a17b17e0e0e1e94c2133c6bb2c5dff62c947 WatchSource:0}: Error finding container 50eb532f2d3f38a8a78d822ae6e3a17b17e0e0e1e94c2133c6bb2c5dff62c947: Status 404 returned error can't find the container with id 50eb532f2d3f38a8a78d822ae6e3a17b17e0e0e1e94c2133c6bb2c5dff62c947
Oct 30 15:11:33.265342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.265316    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-0e87a3"]
Oct 30 15:11:33.388538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.388499    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:33.752575 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:33.752544    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6b9b60_8ec4_41c9_9bb6_1fcedf711d43.slice/crio-ef1aaf62c3db20c486f2e8bf451568203b9a126f393f10bfc6e0725ddaf11e82 WatchSource:0}: Error finding container ef1aaf62c3db20c486f2e8bf451568203b9a126f393f10bfc6e0725ddaf11e82: Status 404 returned error can't find the container with id ef1aaf62c3db20c486f2e8bf451568203b9a126f393f10bfc6e0725ddaf11e82
Oct 30 15:11:33.753059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.753033    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:11:33.755821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.755786    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:33.755949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.755898    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:33.755949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:33.755934    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:11:34.115509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.115412    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-0e87a3" event={"ID":"1420f5fe-20f0-4cce-9e5f-9ee55a326f80","Type":"ContainerStarted","Data":"5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665"}
Oct 30 15:11:34.115509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.115464    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-0e87a3" event={"ID":"1420f5fe-20f0-4cce-9e5f-9ee55a326f80","Type":"ContainerStarted","Data":"50eb532f2d3f38a8a78d822ae6e3a17b17e0e0e1e94c2133c6bb2c5dff62c947"}
Oct 30 15:11:34.118822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.118792    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerStarted","Data":"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"}
Oct 30 15:11:34.118972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.118858    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerStarted","Data":"ef1aaf62c3db20c486f2e8bf451568203b9a126f393f10bfc6e0725ddaf11e82"}
Oct 30 15:11:34.162032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.161974    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9025-0e87a3" podStartSLOduration=2.161958728 podStartE2EDuration="2.161958728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:34.157135856 +0000 UTC m=+1443.599026573" watchObservedRunningTime="2026-10-30 15:11:34.161958728 +0000 UTC m=+1443.603849445"
Oct 30 15:11:34.209568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.209472    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podStartSLOduration=2.209452505 podStartE2EDuration="2.209452505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:34.204772961 +0000 UTC m=+1443.646663675" watchObservedRunningTime="2026-10-30 15:11:34.209452505 +0000 UTC m=+1443.651343223"
Oct 30 15:11:34.392873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.391945    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:34.417033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:34.415916    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:35.116170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.116132    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9025-0e87a3"]
Oct 30 15:11:35.124999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.122917    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9025-0e87a3"]
Oct 30 15:11:35.131521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.131105    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:35.132391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.132364    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:11:35.287108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.287073    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9025-4805f3"]
Oct 30 15:11:35.288013 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.287988    5064 cpu_manager.go:414] "containerMap: removing container" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" containerName="test-curl-9025-0e87a3"
Oct 30 15:11:35.288093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.288015    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" containerName="test-curl-9025-0e87a3"
Oct 30 15:11:35.288290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.288274    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" containerName="test-curl-9025-0e87a3"
Oct 30 15:11:35.293265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.293240    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:35.303953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.299292    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qxl\" (UniqueName: \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\") pod \"test-curl-9025-4805f3\" (UID: \"80e6a1ad-32ec-4467-be63-638485a746e7\") " pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:35.309508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.309479    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-4805f3"]
Oct 30 15:11:35.400529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.400425    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qxl\" (UniqueName: \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\") pod \"test-curl-9025-4805f3\" (UID: \"80e6a1ad-32ec-4467-be63-638485a746e7\") " pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:35.426298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.426271    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qxl\" (UniqueName: \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\") pod \"test-curl-9025-4805f3\" (UID: \"80e6a1ad-32ec-4467-be63-638485a746e7\") " pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:35.625493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.622970    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:35.632521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.632477    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:35.723208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.722175    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\" (UniqueName: \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\") pod \"7ca71dda-80f4-4e81-ba74-02807fec0331\" (UID: \"7ca71dda-80f4-4e81-ba74-02807fec0331\") "
Oct 30 15:11:35.728424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.725116    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w" pod "7ca71dda-80f4-4e81-ba74-02807fec0331" (UID: "7ca71dda-80f4-4e81-ba74-02807fec0331"). InnerVolumeSpecName "kube-api-access-55s8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:35.824272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.824235    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55s8w\" (UniqueName: \"kubernetes.io/projected/7ca71dda-80f4-4e81-ba74-02807fec0331-kube-api-access-55s8w\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:35.954282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:35.954255    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9025-4805f3"]
Oct 30 15:11:35.956088 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:11:35.956016    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e6a1ad_32ec_4467_be63_638485a746e7.slice/crio-ee992e2f2db9f60c11bd70d6442e0193ab5a901c3f27a486d98bd2abae9995e6 WatchSource:0}: Error finding container ee992e2f2db9f60c11bd70d6442e0193ab5a901c3f27a486d98bd2abae9995e6: Status 404 returned error can't find the container with id ee992e2f2db9f60c11bd70d6442e0193ab5a901c3f27a486d98bd2abae9995e6
Oct 30 15:11:36.183699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.168188    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-4805f3" event={"ID":"80e6a1ad-32ec-4467-be63-638485a746e7","Type":"ContainerStarted","Data":"ee992e2f2db9f60c11bd70d6442e0193ab5a901c3f27a486d98bd2abae9995e6"}
Oct 30 15:11:36.183699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.170175    5064 generic.go:356] "Generic (PLEG): container finished" podID="7ca71dda-80f4-4e81-ba74-02807fec0331" containerID="ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6" exitCode=137
Oct 30 15:11:36.183699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.171499    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:36.192188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.186031    5064 scope.go:122] "RemoveContainer" containerID="ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6"
Oct 30 15:11:36.197729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.197690    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9025-0e87a3" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" containerName="test-curl-9025-0e87a3" containerID="cri-o://5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665" gracePeriod=2
Oct 30 15:11:36.215514 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:36.215195    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-0e87a3\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:36.222957 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.222615    5064 scope.go:122] "RemoveContainer" containerID="ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6"
Oct 30 15:11:36.223265 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:36.223165    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6\": container with ID starting with ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6 not found: ID does not exist" containerID="ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6"
Oct 30 15:11:36.223265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.223205    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6"} err="failed to get container status \"ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6\": rpc error: code = NotFound desc = could not find container \"ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6\": container with ID starting with ed75ed0863c53cdbc06c7eb3cc20e9d7d2be167686a84704ad9156d9246d2dd6 not found: ID does not exist"
Oct 30 15:11:36.230712 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:36.230492    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-c185ce\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" pod="opendatahub/test-curl-9025-c185ce"
Oct 30 15:11:36.251923 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:36.251854    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-0e87a3\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:36.727529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:36.727496    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca71dda-80f4-4e81-ba74-02807fec0331" path="/var/lib/kubelet/pods/7ca71dda-80f4-4e81-ba74-02807fec0331/volumes"
Oct 30 15:11:37.247854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:37.245167    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9025-4805f3" event={"ID":"80e6a1ad-32ec-4467-be63-638485a746e7","Type":"ContainerStarted","Data":"aea0a78a22a54ede97a06d9a9f61cd61732f60b778e3a8ab710c7dec4edbe16d"}
Oct 30 15:11:37.307854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:37.297431    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9025-4805f3" podStartSLOduration=2.297414521 podStartE2EDuration="2.297414521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:37.295858262 +0000 UTC m=+1446.737748981" watchObservedRunningTime="2026-10-30 15:11:37.297414521 +0000 UTC m=+1446.739305239"
Oct 30 15:11:38.549489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.549458    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:38.634982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.634947    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9025-4805f3"]
Oct 30 15:11:38.640575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.640543    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9025-4805f3"]
Oct 30 15:11:38.722327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.722293    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\" (UniqueName: \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\") pod \"1420f5fe-20f0-4cce-9e5f-9ee55a326f80\" (UID: \"1420f5fe-20f0-4cce-9e5f-9ee55a326f80\") "
Oct 30 15:11:38.724554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.724528    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5" pod "1420f5fe-20f0-4cce-9e5f-9ee55a326f80" (UID: "1420f5fe-20f0-4cce-9e5f-9ee55a326f80"). InnerVolumeSpecName "kube-api-access-2h5s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:38.728785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.728758    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" path="/var/lib/kubelet/pods/1420f5fe-20f0-4cce-9e5f-9ee55a326f80/volumes"
Oct 30 15:11:38.825703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:38.825569    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5s5\" (UniqueName: \"kubernetes.io/projected/1420f5fe-20f0-4cce-9e5f-9ee55a326f80-kube-api-access-2h5s5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:39.316805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.316757    5064 generic.go:356] "Generic (PLEG): container finished" podID="1420f5fe-20f0-4cce-9e5f-9ee55a326f80" containerID="5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665" exitCode=137
Oct 30 15:11:39.317138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.317113    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-0e87a3"
Oct 30 15:11:39.317919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.317646    5064 scope.go:122] "RemoveContainer" containerID="5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665"
Oct 30 15:11:39.318215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.318171    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9025-4805f3" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" containerName="test-curl-9025-4805f3" containerID="cri-o://aea0a78a22a54ede97a06d9a9f61cd61732f60b778e3a8ab710c7dec4edbe16d" gracePeriod=2
Oct 30 15:11:39.330144 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.330115    5064 scope.go:122] "RemoveContainer" containerID="5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665"
Oct 30 15:11:39.330532 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:39.330506    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665\": container with ID starting with 5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665 not found: ID does not exist" containerID="5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665"
Oct 30 15:11:39.330631 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:39.330543    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665"} err="failed to get container status \"5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665\": rpc error: code = NotFound desc = could not find container \"5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665\": container with ID starting with 5b281c95d1a577ff0495bc8426824dad5db8c5a1bfa024b8fc39b4f4bcdc2665 not found: ID does not exist"
Oct 30 15:11:41.478481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:41.478440    5064 generic.go:356] "Generic (PLEG): container finished" podID="80e6a1ad-32ec-4467-be63-638485a746e7" containerID="aea0a78a22a54ede97a06d9a9f61cd61732f60b778e3a8ab710c7dec4edbe16d" exitCode=137
Oct 30 15:11:41.573468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:41.573399    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:41.687626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:41.687592    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\" (UniqueName: \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\") pod \"80e6a1ad-32ec-4467-be63-638485a746e7\" (UID: \"80e6a1ad-32ec-4467-be63-638485a746e7\") "
Oct 30 15:11:41.689887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:41.689818    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl" pod "80e6a1ad-32ec-4467-be63-638485a746e7" (UID: "80e6a1ad-32ec-4467-be63-638485a746e7"). InnerVolumeSpecName "kube-api-access-x2qxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:11:41.789429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:41.789393    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2qxl\" (UniqueName: \"kubernetes.io/projected/80e6a1ad-32ec-4467-be63-638485a746e7-kube-api-access-x2qxl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:11:42.488722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:42.488689    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:42.488722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:42.488709    5064 scope.go:122] "RemoveContainer" containerID="aea0a78a22a54ede97a06d9a9f61cd61732f60b778e3a8ab710c7dec4edbe16d"
Oct 30 15:11:42.513685 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:42.513644    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9025-4805f3\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" pod="opendatahub/test-curl-9025-4805f3"
Oct 30 15:11:42.725640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:11:42.725600    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" path="/var/lib/kubelet/pods/80e6a1ad-32ec-4467-be63-638485a746e7/volumes"
Oct 30 15:11:59.632293 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:11:59.632255    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:12:05.401433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.401400    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:12:05.402456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.402436    5064 cpu_manager.go:414] "containerMap: removing container" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" containerName="test-curl-9025-4805f3"
Oct 30 15:12:05.402456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.402456    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" containerName="test-curl-9025-4805f3"
Oct 30 15:12:05.402606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.402587    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e6a1ad-32ec-4467-be63-638485a746e7" containerName="test-curl-9025-4805f3"
Oct 30 15:12:05.407683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.407663    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.431764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.431714    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:12:05.435663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.435632    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-147bdb-serving-cert\""
Oct 30 15:12:05.516667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.516632    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.516822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.516696    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.516964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.516916    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.516964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.516957    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.618542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.618501    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.618726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.618637    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.618795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.618779    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.618851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.618809    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.619460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.619436    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.621110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.621091    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.621244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.621221    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.706860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.699913    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\") pod \"maas-api-e2e-api-a-147bdb-864c69d766-9sh5g\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:05.728240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:05.728144    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:06.170964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:06.167671    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:12:07.102921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:07.102767    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" event={"ID":"93ea3586-be8b-4246-a2b7-42f3c399883a","Type":"ContainerStarted","Data":"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"}
Oct 30 15:12:07.102921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:07.102877    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" event={"ID":"93ea3586-be8b-4246-a2b7-42f3c399883a","Type":"ContainerStarted","Data":"513875e6c6e16a444dd74f70e092dd2b9c892352334bde7cbc6e7ba60c718119"}
Oct 30 15:12:07.103998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:07.103966    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:07.151648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:07.151589    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" podStartSLOduration=1.654200897 podStartE2EDuration="2.151575544s" totalImagesPullingTime="497.374647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:07.149513252 +0000 UTC m=+1476.591403979" watchObservedRunningTime="2026-10-30 15:12:07.151575544 +0000 UTC m=+1476.593466260"
Oct 30 15:12:11.022181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.022081    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:12:11.022998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.022959    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy" containerID="cri-o://cd2caf0c32ac91c975630e04e0b59e6ac9c93bba3365d81676d4a534ee9af2a6" gracePeriod=30
Oct 30 15:12:11.047988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.047915    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:12:11.048221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.048195    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy" containerID="cri-o://32f8a46c427807d14cdf1dad9fb7f7b1f732ab67a15fc0c5d77fb0a1062acf82" gracePeriod=30
Oct 30 15:12:11.051672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.051642    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:12:11.051948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.051901    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy" containerID="cri-o://5ef376a19a4bad988f70afa4bb48303b17771ffe73eeb964f6c25bf5a1de8d03" gracePeriod=30
Oct 30 15:12:11.084655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.084611    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:12:11.085045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:11.085012    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy" containerID="cri-o://e23f8f1d050138e050b9021024c260d5c7f7609ca6247662d596554e78cf0dc1" gracePeriod=30
Oct 30 15:12:14.123780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:14.123748    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.250579    5064 generic.go:356] "Generic (PLEG): container finished" podID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerID="32f8a46c427807d14cdf1dad9fb7f7b1f732ab67a15fc0c5d77fb0a1062acf82" exitCode=0
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.250697    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" event={"ID":"5b453573-d86d-43c2-81d0-b7c74aee2706","Type":"ContainerDied","Data":"32f8a46c427807d14cdf1dad9fb7f7b1f732ab67a15fc0c5d77fb0a1062acf82"}
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.252937    5064 generic.go:356] "Generic (PLEG): container finished" podID="90ea1616-83a5-4876-bfee-915cf56206ba" containerID="e23f8f1d050138e050b9021024c260d5c7f7609ca6247662d596554e78cf0dc1" exitCode=0
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.253047    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" event={"ID":"90ea1616-83a5-4876-bfee-915cf56206ba","Type":"ContainerDied","Data":"e23f8f1d050138e050b9021024c260d5c7f7609ca6247662d596554e78cf0dc1"}
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.255341    5064 generic.go:356] "Generic (PLEG): container finished" podID="df46a028-001f-4536-b3c3-b324b54634cd" containerID="5ef376a19a4bad988f70afa4bb48303b17771ffe73eeb964f6c25bf5a1de8d03" exitCode=0
Oct 30 15:12:16.257570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.255433    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" event={"ID":"df46a028-001f-4536-b3c3-b324b54634cd","Type":"ContainerDied","Data":"5ef376a19a4bad988f70afa4bb48303b17771ffe73eeb964f6c25bf5a1de8d03"}
Oct 30 15:12:16.258273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.257867    5064 generic.go:356] "Generic (PLEG): container finished" podID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerID="cd2caf0c32ac91c975630e04e0b59e6ac9c93bba3365d81676d4a534ee9af2a6" exitCode=0
Oct 30 15:12:16.258273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.257916    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" event={"ID":"874909f2-47cb-4dee-ba41-0744403bc5e9","Type":"ContainerDied","Data":"cd2caf0c32ac91c975630e04e0b59e6ac9c93bba3365d81676d4a534ee9af2a6"}
Oct 30 15:12:16.432044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.432004    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:12:16.447975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.447940    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448114    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448201    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448269    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448363    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448419    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448549    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448631 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448593    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.448724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.448707    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\") pod \"874909f2-47cb-4dee-ba41-0744403bc5e9\" (UID: \"874909f2-47cb-4dee-ba41-0744403bc5e9\") "
Oct 30 15:12:16.451053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.450231    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.451053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.450526    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:12:16.451053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.450751    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.451527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.451505    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.452142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.452100    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.453337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.453310    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.454898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.454876    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.455192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.455173    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "kube-api-access-hk9r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.456768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.456742    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo" pod "874909f2-47cb-4dee-ba41-0744403bc5e9" (UID: "874909f2-47cb-4dee-ba41-0744403bc5e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:12:16.461210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.461185    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:12:16.485365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.485336    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:12:16.490245 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.490219    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:12:16.550711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550675    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.550964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550733    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.550964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550789    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.550964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550886    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.550964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550951    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.551169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.550991    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.551169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551090    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.551169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551112    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.551588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551168    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.551588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551238    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.552034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551708    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.552034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.551984    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552233    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552278    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552338    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552365    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552415    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552495    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552549    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552581    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552624    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552672    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552698    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552724    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552812    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.552879    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.555222    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.555912    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.558649    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.558682    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\") pod \"5b453573-d86d-43c2-81d0-b7c74aee2706\" (UID: \"5b453573-d86d-43c2-81d0-b7c74aee2706\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.558744    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.558806    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\") pod \"90ea1616-83a5-4876-bfee-915cf56206ba\" (UID: \"90ea1616-83a5-4876-bfee-915cf56206ba\") "
Oct 30 15:12:16.558904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.558841    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\") pod \"df46a028-001f-4536-b3c3-b324b54634cd\" (UID: \"df46a028-001f-4536-b3c3-b324b54634cd\") "
Oct 30 15:12:16.560456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560431    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9r8\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-kube-api-access-hk9r8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560463    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560479    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874909f2-47cb-4dee-ba41-0744403bc5e9-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560492    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560505    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560519    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560531    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560544    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560559    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.560578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560571    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560586    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560601    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560615    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560627    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560641    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874909f2-47cb-4dee-ba41-0744403bc5e9-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.560859    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.561112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.561047    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.561502    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.561687    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.561774    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.562016    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.562186    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.562436    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:12:16.563183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.562689    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.563614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.563452    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:12:16.564237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.564212    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.564482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.564458    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.565968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.565941    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.567305 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.567278    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.567751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.567724    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:12:16.568275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.568256    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh" pod "90ea1616-83a5-4876-bfee-915cf56206ba" (UID: "90ea1616-83a5-4876-bfee-915cf56206ba"). InnerVolumeSpecName "kube-api-access-x8tbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.568387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.568285    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.568437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.568392    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:12:16.568515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.568495    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "kube-api-access-gn4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.568651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.568629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token" pod "df46a028-001f-4536-b3c3-b324b54634cd" (UID: "df46a028-001f-4536-b3c3-b324b54634cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.570565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.570102    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g" pod "5b453573-d86d-43c2-81d0-b7c74aee2706" (UID: "5b453573-d86d-43c2-81d0-b7c74aee2706"). InnerVolumeSpecName "kube-api-access-pkb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.662931    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ea1616-83a5-4876-bfee-915cf56206ba-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.662964    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.662980    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkb8g\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-kube-api-access-pkb8g\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.662995    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663012    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df46a028-001f-4536-b3c3-b324b54634cd-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663026    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8tbh\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-kube-api-access-x8tbh\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663039    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663052    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663066    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663078    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ea1616-83a5-4876-bfee-915cf56206ba-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663088    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663099    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ea1616-83a5-4876-bfee-915cf56206ba-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663111    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b453573-d86d-43c2-81d0-b7c74aee2706-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663123    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b453573-d86d-43c2-81d0-b7c74aee2706-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663135    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b453573-d86d-43c2-81d0-b7c74aee2706-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663146    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663154    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ea1616-83a5-4876-bfee-915cf56206ba-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663162    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663175    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df46a028-001f-4536-b3c3-b324b54634cd-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663186    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df46a028-001f-4536-b3c3-b324b54634cd-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:16.663218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:16.663195    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn4sv\" (UniqueName: \"kubernetes.io/projected/df46a028-001f-4536-b3c3-b324b54634cd-kube-api-access-gn4sv\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:12:17.268468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.268436    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"
Oct 30 15:12:17.268468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.268450    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq" event={"ID":"5b453573-d86d-43c2-81d0-b7c74aee2706","Type":"ContainerDied","Data":"a8083f74c0da38b103f3b3a1afdced24c4c3655f81e135afe6d208c9c4582551"}
Oct 30 15:12:17.269036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.268523    5064 scope.go:122] "RemoveContainer" containerID="32f8a46c427807d14cdf1dad9fb7f7b1f732ab67a15fc0c5d77fb0a1062acf82"
Oct 30 15:12:17.271090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.270985    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"
Oct 30 15:12:17.271523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.271501    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v" event={"ID":"90ea1616-83a5-4876-bfee-915cf56206ba","Type":"ContainerDied","Data":"d939212beb414cbf0ef668b1f907d8271c61104838ab5815aa452220ff5dd291"}
Oct 30 15:12:17.272780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.272760    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"
Oct 30 15:12:17.272932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.272763    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx" event={"ID":"df46a028-001f-4536-b3c3-b324b54634cd","Type":"ContainerDied","Data":"ef55c5bba3208ff36e02933c3178a903cb603ae5a04aa326c019b197e5b760cf"}
Oct 30 15:12:17.274477 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.274455    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"
Oct 30 15:12:17.274573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.274474    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2" event={"ID":"874909f2-47cb-4dee-ba41-0744403bc5e9","Type":"ContainerDied","Data":"62d2242a5d2c6527a315cbaa837275b47312befe0257a5e17273b42915ac278f"}
Oct 30 15:12:17.282585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.282557    5064 scope.go:122] "RemoveContainer" containerID="e23f8f1d050138e050b9021024c260d5c7f7609ca6247662d596554e78cf0dc1"
Oct 30 15:12:17.293181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.293162    5064 scope.go:122] "RemoveContainer" containerID="5ef376a19a4bad988f70afa4bb48303b17771ffe73eeb964f6c25bf5a1de8d03"
Oct 30 15:12:17.304193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.304173    5064 scope.go:122] "RemoveContainer" containerID="cd2caf0c32ac91c975630e04e0b59e6ac9c93bba3365d81676d4a534ee9af2a6"
Oct 30 15:12:17.328298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.328267    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:12:17.342224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.342187    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-9r7x2"]
Oct 30 15:12:17.370936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.370900    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:12:17.379669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.379605    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-jqjwx"]
Oct 30 15:12:17.431374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.431335    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:12:17.439142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.439086    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k49nq"]
Oct 30 15:12:17.478651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.478610    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:12:17.482522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:17.482496    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-66x8v"]
Oct 30 15:12:18.725164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:18.725131    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" path="/var/lib/kubelet/pods/5b453573-d86d-43c2-81d0-b7c74aee2706/volumes"
Oct 30 15:12:18.725722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:18.725706    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" path="/var/lib/kubelet/pods/874909f2-47cb-4dee-ba41-0744403bc5e9/volumes"
Oct 30 15:12:18.726239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:18.726223    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" path="/var/lib/kubelet/pods/90ea1616-83a5-4876-bfee-915cf56206ba/volumes"
Oct 30 15:12:18.727110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:18.727093    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df46a028-001f-4536-b3c3-b324b54634cd" path="/var/lib/kubelet/pods/df46a028-001f-4536-b3c3-b324b54634cd/volumes"
Oct 30 15:12:19.155179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.155137    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:12:19.156811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156782    5064 cpu_manager.go:414] "containerMap: removing container" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy"
Oct 30 15:12:19.156811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156811    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy"
Oct 30 15:12:19.157023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156867    5064 cpu_manager.go:414] "containerMap: removing container" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy"
Oct 30 15:12:19.157023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156876    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy"
Oct 30 15:12:19.157023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156971    5064 cpu_manager.go:414] "containerMap: removing container" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy"
Oct 30 15:12:19.157023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.156980    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy"
Oct 30 15:12:19.157170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157027    5064 cpu_manager.go:414] "containerMap: removing container" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy"
Oct 30 15:12:19.157170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157035    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy"
Oct 30 15:12:19.157574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157554    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="874909f2-47cb-4dee-ba41-0744403bc5e9" containerName="istio-proxy"
Oct 30 15:12:19.157648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157584    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b453573-d86d-43c2-81d0-b7c74aee2706" containerName="istio-proxy"
Oct 30 15:12:19.157648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157596    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="df46a028-001f-4536-b3c3-b324b54634cd" containerName="istio-proxy"
Oct 30 15:12:19.157648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.157604    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ea1616-83a5-4876-bfee-915cf56206ba" containerName="istio-proxy"
Oct 30 15:12:19.163423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.163402    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.171347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.171308    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:12:19.192909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.183306    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24fff1-serving-cert\""
Oct 30 15:12:19.224435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.224394    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.224671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.224444    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898n8\" (UniqueName: \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.224754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.224695    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.225039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.224993    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.326299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.326255    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.326479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.326378    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.326617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.326601    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.326684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.326630    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898n8\" (UniqueName: \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.327281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.327251    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.328959 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.328938    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.329086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.329066    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.352567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.352538    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-898n8\" (UniqueName: \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\") pod \"maas-api-e2e-inf-b-24fff1-f47959d89-dppmv\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.489472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.489370    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:19.823887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:19.823846    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:12:19.824798 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:12:19.824770    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cbf1a8_167f_48bc_9b5b_c4113791a223.slice/crio-14c28174cd171085ae68fe7e936f6d7345299812f6cb8605a38fc524e2998ac2 WatchSource:0}: Error finding container 14c28174cd171085ae68fe7e936f6d7345299812f6cb8605a38fc524e2998ac2: Status 404 returned error can't find the container with id 14c28174cd171085ae68fe7e936f6d7345299812f6cb8605a38fc524e2998ac2
Oct 30 15:12:20.303667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.303630    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" event={"ID":"91cbf1a8-167f-48bc-9b5b-c4113791a223","Type":"ContainerStarted","Data":"14c28174cd171085ae68fe7e936f6d7345299812f6cb8605a38fc524e2998ac2"}
Oct 30 15:12:20.802885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.802851    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:12:20.807778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.807757    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.823822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.823794    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:12:20.832153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.828381    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.840589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.834113    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:12:20.840589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.834910    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-216204-openshift-default-dockercfg-ld9nk\""
Oct 30 15:12:20.855718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.855684    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.879980    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880067    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880103    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880221    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880346    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880510    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880615    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880726    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbm7\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.881627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.880850    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.982488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982455    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.982660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982502    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.982660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982580    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.982660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982610    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.982660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982639    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.982939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982685    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.982939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982724    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.982939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.982804    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsbl\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983002    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983043    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983110    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983265    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983316    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983316    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983391    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983441    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983466    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983516    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983553    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbm7\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983645    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:20.983718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983679    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983703    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.983941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.983924    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.985290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.985268    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:20.986034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:20.986007    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:21.008484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.008441    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:21.008627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.008581    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbm7\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-6kxdl\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:21.085290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085200    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085281    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085396    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085472    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085539    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085601    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085636    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsbl\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085689    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085707    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085735    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.085990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.085792    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.086092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.086071    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.086216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.086193    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.086288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.086202    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.087902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.087877    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.088386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.088369    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.111261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.111232    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.111416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.111326    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsbl\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-xmcfc\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.125320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.125273    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:21.163028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.162991    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:21.316020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.315981    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" event={"ID":"91cbf1a8-167f-48bc-9b5b-c4113791a223","Type":"ContainerStarted","Data":"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"}
Oct 30 15:12:21.316199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.316108    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:21.367239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.367186    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" podStartSLOduration=1.939438706 podStartE2EDuration="2.367170178s" totalImagesPullingTime="427.731472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:21.362727982 +0000 UTC m=+1490.804618698" watchObservedRunningTime="2026-10-30 15:12:21.367170178 +0000 UTC m=+1490.809060895"
Oct 30 15:12:21.455014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.454989    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:12:21.458879 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:12:21.457626    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abee056_d875_4f97_a6a9_c4b9a8269c90.slice/crio-396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa WatchSource:0}: Error finding container 396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa: Status 404 returned error can't find the container with id 396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa
Oct 30 15:12:21.461774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.461732    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:21.461900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.461840    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:21.461900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.461893    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:21.543287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.543254    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:12:21.560916 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:12:21.560815    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b28df9_cdce_4ba2_b170_390a0b2d229d.slice/crio-6f09f902496bf201efe60b7daa2431b3de32908c3ca3eba09369b8d3fd892f5a WatchSource:0}: Error finding container 6f09f902496bf201efe60b7daa2431b3de32908c3ca3eba09369b8d3fd892f5a: Status 404 returned error can't find the container with id 6f09f902496bf201efe60b7daa2431b3de32908c3ca3eba09369b8d3fd892f5a
Oct 30 15:12:21.567867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.565997    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:21.567867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.566106    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:21.567867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:21.566197    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:12:22.348377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:22.348332    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" event={"ID":"b6b28df9-cdce-4ba2-b170-390a0b2d229d","Type":"ContainerStarted","Data":"adac9558f9dc8ed29348504d322b23f1758e369b81d64c1578d59e09a04309a9"}
Oct 30 15:12:22.348894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:22.348439    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" event={"ID":"b6b28df9-cdce-4ba2-b170-390a0b2d229d","Type":"ContainerStarted","Data":"6f09f902496bf201efe60b7daa2431b3de32908c3ca3eba09369b8d3fd892f5a"}
Oct 30 15:12:22.350277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:22.350233    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" event={"ID":"4abee056-d875-4f97-a6a9-c4b9a8269c90","Type":"ContainerStarted","Data":"e38b1a15bbcb5b0f93d1ac63e5efd4a41ca3b19ba8c1783655267c7159f94bde"}
Oct 30 15:12:22.350429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:22.350293    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" event={"ID":"4abee056-d875-4f97-a6a9-c4b9a8269c90","Type":"ContainerStarted","Data":"396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa"}
Oct 30 15:12:22.402985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:22.401434    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podStartSLOduration=2.40141659 podStartE2EDuration="2.40141659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:22.4001749 +0000 UTC m=+1491.842065626" watchObservedRunningTime="2026-10-30 15:12:22.40141659 +0000 UTC m=+1491.843307306"
Oct 30 15:12:23.152855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.128471    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:23.166951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.165533    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:23.166951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.165707    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:23.166951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.165761    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:23.166951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.165709    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:23.166951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:23.166069    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:24.131085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:24.131048    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:24.132160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:24.131741    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:24.166856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:24.166035    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:24.166856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:24.166103    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:25.139855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:25.139251    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:25.139855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:25.139317    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:25.165478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:25.165283    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:25.165478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:25.165350    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:26.140856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:26.137070    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:26.140856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:26.137131    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:26.207857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:26.206904    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:26.207857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:26.206964    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:27.128382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.128336    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:27.128693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.128663    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:27.168453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.168410    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:27.169089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.169037    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:27.367852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.367429    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:12:27.451945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:27.451327    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podStartSLOduration=7.420583683 podStartE2EDuration="7.420583683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:22.46926566 +0000 UTC m=+1491.911156378" watchObservedRunningTime="2026-10-30 15:12:27.420583683 +0000 UTC m=+1496.862474401"
Oct 30 15:12:28.138855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:28.133274    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 15:12:28.138855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:28.133362    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 15:12:28.163613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:28.163563    5064 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 15:12:28.163869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:28.163647    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 15:12:29.204179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.204141    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:29.204880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.204856    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:29.205694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.205674    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:12:29.260602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.260570    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:29.261002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.260966    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:29.261886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:29.261865    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:12:29.632328 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:12:29.632290    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:12:30.229641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.229605    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:12:30.235897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.235871    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.259900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.259868    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:12:30.272877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.271889    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1b7814-serving-cert\""
Oct 30 15:12:30.313861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.313788    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfh47\" (UniqueName: \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.314035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.313946    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.314093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.314069    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.314260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.314229    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.416090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.416052    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.416264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.416164    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.416347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.416326    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh47\" (UniqueName: \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.416431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.416415    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.416892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.416855    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.418870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.418847    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.418945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.418894    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.442134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.442093    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfh47\" (UniqueName: \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\") pod \"maas-api-e2e-api-a-1b7814-66c96984b5-5wt85\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.575908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.573203    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:30.936091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.936063    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:12:30.936302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.936119    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:12:30.950364 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:30.950342    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:12:30.951037 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:12:30.951015    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f1b8ed_f8e8_4e4c_ad50_88a0249ecb0e.slice/crio-677823a03b862255af4706b391693d959b18254be8386296fd95b23c98cc9cca WatchSource:0}: Error finding container 677823a03b862255af4706b391693d959b18254be8386296fd95b23c98cc9cca: Status 404 returned error can't find the container with id 677823a03b862255af4706b391693d959b18254be8386296fd95b23c98cc9cca
Oct 30 15:12:31.454821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:31.454783    5064 scope.go:122] "RemoveContainer" containerID="0bdaa1279f78fc3a6c84b8aa25b2aad7c1da4a3d9cedc6da6256620cd2bfc4f1"
Oct 30 15:12:31.660221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:31.660183    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" event={"ID":"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e","Type":"ContainerStarted","Data":"b1e074c405a483dc83b899f9dd33ccf6366a253609bd4d1b139b61b4ee168c1f"}
Oct 30 15:12:31.660396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:31.660244    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" event={"ID":"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e","Type":"ContainerStarted","Data":"677823a03b862255af4706b391693d959b18254be8386296fd95b23c98cc9cca"}
Oct 30 15:12:31.660396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:31.660351    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:31.706818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:31.706713    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" podStartSLOduration=1.1447895670000001 podStartE2EDuration="1.706699659s" totalImagesPullingTime="561.910092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:31.706102331 +0000 UTC m=+1501.147993047" watchObservedRunningTime="2026-10-30 15:12:31.706699659 +0000 UTC m=+1501.148590376"
Oct 30 15:12:37.674965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:37.674931    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:12:41.759463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.759425    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:12:41.765729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.765703    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.787928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.787885    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:12:41.788577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.788547    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7acff45-serving-cert\""
Oct 30 15:12:41.860508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.860465    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.860734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.860538    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46fm\" (UniqueName: \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.860911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.860867    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.861056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.861031    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.962814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.962777    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.963044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.963028    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.963106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.963079    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46fm\" (UniqueName: \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.963160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.963141    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.963715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.963692    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.965506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.965481    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:41.965607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:41.965583    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:42.037035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:42.033285    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46fm\" (UniqueName: \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:42.095862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:42.093307    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:42.472566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:42.472531    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:12:42.472705 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:12:42.472675    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238273c2_e6a7_43a2_b5c1_2eb2d3d378bb.slice/crio-0923a5500e5f9594dbb0b631ad50be8a7d8bfc949a4cde591e287d60399dd69e WatchSource:0}: Error finding container 0923a5500e5f9594dbb0b631ad50be8a7d8bfc949a4cde591e287d60399dd69e: Status 404 returned error can't find the container with id 0923a5500e5f9594dbb0b631ad50be8a7d8bfc949a4cde591e287d60399dd69e
Oct 30 15:12:42.789055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:42.788962    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" event={"ID":"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb","Type":"ContainerStarted","Data":"0923a5500e5f9594dbb0b631ad50be8a7d8bfc949a4cde591e287d60399dd69e"}
Oct 30 15:12:43.798648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:43.798601    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" event={"ID":"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb","Type":"ContainerStarted","Data":"a384aea214745189748e75709fa7c4367db3c71ce81061480f4dc5f56390cbf1"}
Oct 30 15:12:43.799135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:43.798694    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:43.869377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:43.869316    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" podStartSLOduration=2.13943554 podStartE2EDuration="2.869299678s" totalImagesPullingTime="729.864138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:43.86515439 +0000 UTC m=+1513.307045107" watchObservedRunningTime="2026-10-30 15:12:43.869299678 +0000 UTC m=+1513.311190433"
Oct 30 15:12:49.812636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:12:49.812602    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:12:59.635633 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:12:59.635594    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:13:03.336626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.336588    5064 generic.go:356] "Generic (PLEG): container finished" podID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerID="871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19" exitCode=0
Oct 30 15:13:03.338390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.336708    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerDied","Data":"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"}
Oct 30 15:13:03.338390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.337305    5064 scope.go:122] "RemoveContainer" containerID="871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"
Oct 30 15:13:03.346319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.346059    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:03.346319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.346161    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:03.346319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:03.346209    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:04.360128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:04.360092    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerStarted","Data":"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"}
Oct 30 15:13:05.324777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:05.302993    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:05.324777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:05.303063    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:05.346697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:05.346664    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:13:06.141556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:06.141504    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:06.142102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:06.141579    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:07.154854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:07.144366    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:07.154854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:07.144435    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:07.541749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:07.541662    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:13:08.141346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:08.141307    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:08.141567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:08.141363    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:09.148854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:09.147940    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:09.148854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:09.148010    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:10.160977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:10.148325    5064 patch_prober.go:29] interesting pod/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 15:13:10.160977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:10.148410    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 15:13:11.146763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:11.146727    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:13:11.148174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:11.148147    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:13:18.851752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.851720    5064 generic.go:356] "Generic (PLEG): container finished" podID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerID="c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1" exitCode=0
Oct 30 15:13:18.852278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.851798    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerDied","Data":"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"}
Oct 30 15:13:18.852441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.852423    5064 scope.go:122] "RemoveContainer" containerID="c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"
Oct 30 15:13:18.855320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.855288    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:18.855422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.855362    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:18.855422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:18.855402    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:19.862236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:19.862182    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerStarted","Data":"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"}
Oct 30 15:13:20.131138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:20.131099    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:13:20.389732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:20.389647    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:13:20.390909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:20.390878    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:20.391038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:20.390935    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:21.389176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:21.389124    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:21.389581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:21.389190    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:22.389612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.389568    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:22.390051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.389645    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:22.799004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.798912    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:13:22.803422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.803394    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.865382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.865351    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-47bcfe-openshift-default-dockercfg-r8sfk\""
Oct 30 15:13:22.878960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.878915    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:13:22.982928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.982877    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.983231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.983207    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.983374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.983357    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.983525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.983508    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.983728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.983710    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:22.983991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:22.983971    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.001068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.001030    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.001342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.001322    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9t5\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.001775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.001755    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149335    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149436    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149466    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149502    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149554    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149645    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149768    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.149854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149850    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.150606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.149887    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9t5\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.276855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.276079    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.303863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.288279    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.303863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.288787    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.303863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.289905    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.303863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.296244    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.339017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.338984    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9t5\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.393923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.390121    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:23.393923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.390179    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:23.433560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.433483    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.434787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.434731    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.436893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.436842    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\") pod \"e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") " pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.488319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.488020    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:23.850646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.850611    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:13:23.852715 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:13:23.852675    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd26fed_403a_410f_9bd6_daec9ae621b9.slice/crio-0b51da3831c51c7f90ff2a47027857d8f88df5b52e9be8c3ac41aed64d4c5712 WatchSource:0}: Error finding container 0b51da3831c51c7f90ff2a47027857d8f88df5b52e9be8c3ac41aed64d4c5712: Status 404 returned error can't find the container with id 0b51da3831c51c7f90ff2a47027857d8f88df5b52e9be8c3ac41aed64d4c5712
Oct 30 15:13:23.859447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.859395    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:23.859586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.859501    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:23.859586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.859569    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:13:23.906654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:23.906617    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerStarted","Data":"0b51da3831c51c7f90ff2a47027857d8f88df5b52e9be8c3ac41aed64d4c5712"}
Oct 30 15:13:24.402863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:24.393818    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:24.402863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:24.393922    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:24.939955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:24.939906    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerStarted","Data":"0df976cd18230e7e3367b2fb03c8d5ece89c9136ce2194d30eac6d8b8bc9e83e"}
Oct 30 15:13:25.005665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.005609    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podStartSLOduration=3.00559352 podStartE2EDuration="3.00559352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:25.002194446 +0000 UTC m=+1554.444085163" watchObservedRunningTime="2026-10-30 15:13:25.00559352 +0000 UTC m=+1554.447484239"
Oct 30 15:13:25.388977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.388939    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 15:13:25.389155 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.388998    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 15:13:25.491010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.490972    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:25.497019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.496988    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:25.951387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.951351    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:25.952462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:25.952440    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:13:26.485966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:26.485932    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:13:26.488001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:26.487976    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:13:30.349419 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:30.349374    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:13:37.888164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:37.888122    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9013-987220"]
Oct 30 15:13:37.892711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:37.892678    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:37.915979 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:37.915944    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9013-987220"]
Oct 30 15:13:37.923247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:37.923206    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\""
Oct 30 15:13:38.017711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:38.017674    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbr9\" (UniqueName: \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\") pod \"test-curl-9013-987220\" (UID: \"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2\") " pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:38.120004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:38.119961    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbr9\" (UniqueName: \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\") pod \"test-curl-9013-987220\" (UID: \"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2\") " pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:38.157184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:38.157106    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbr9\" (UniqueName: \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\") pod \"test-curl-9013-987220\" (UID: \"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2\") " pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:38.206223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:38.206185    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:38.515485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:38.515440    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9013-987220"]
Oct 30 15:13:38.516291 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:13:38.516262    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aa2543a_c4c0_48b2_8b9d_2de285a9c2b2.slice/crio-d09e83097a85125fee4c686e75b0b2704b42242e02715a221b4ffba9f101706d WatchSource:0}: Error finding container d09e83097a85125fee4c686e75b0b2704b42242e02715a221b4ffba9f101706d: Status 404 returned error can't find the container with id d09e83097a85125fee4c686e75b0b2704b42242e02715a221b4ffba9f101706d
Oct 30 15:13:39.502292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:39.502250    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9013-987220" event={"ID":"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2","Type":"ContainerStarted","Data":"a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa"}
Oct 30 15:13:39.502292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:39.502303    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9013-987220" event={"ID":"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2","Type":"ContainerStarted","Data":"d09e83097a85125fee4c686e75b0b2704b42242e02715a221b4ffba9f101706d"}
Oct 30 15:13:39.548168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:39.548104    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9013-987220" podStartSLOduration=2.548087731 podStartE2EDuration="2.548087731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:39.543685378 +0000 UTC m=+1568.985576095" watchObservedRunningTime="2026-10-30 15:13:39.548087731 +0000 UTC m=+1568.989978447"
Oct 30 15:13:40.511060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:40.511023    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9013-987220"]
Oct 30 15:13:40.517032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:40.516999    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9013-987220"]
Oct 30 15:13:41.517906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:41.517823    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9013-987220" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" containerName="test-curl-9013-987220" containerID="cri-o://a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa" gracePeriod=2
Oct 30 15:13:43.776442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:43.776373    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:43.853977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:43.853940    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\" (UniqueName: \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\") pod \"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2\" (UID: \"7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2\") "
Oct 30 15:13:43.856108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:43.856078    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9" pod "7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" (UID: "7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2"). InnerVolumeSpecName "kube-api-access-6pbr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:13:43.956081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:43.956042    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pbr9\" (UniqueName: \"kubernetes.io/projected/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2-kube-api-access-6pbr9\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:13:44.555647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.555608    5064 generic.go:356] "Generic (PLEG): container finished" podID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" containerID="a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa" exitCode=137
Oct 30 15:13:44.555872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.555658    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:44.555872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.555714    5064 scope.go:122] "RemoveContainer" containerID="a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa"
Oct 30 15:13:44.566537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.566515    5064 scope.go:122] "RemoveContainer" containerID="a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa"
Oct 30 15:13:44.566866 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:44.566842    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa\": container with ID starting with a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa not found: ID does not exist" containerID="a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa"
Oct 30 15:13:44.566941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.566881    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa"} err="failed to get container status \"a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa\": rpc error: code = NotFound desc = could not find container \"a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa\": container with ID starting with a28e7df5e31643c57006cc4c7bfd10e90e6bf9eecde6d6c9243d5c1f0b75fafa not found: ID does not exist"
Oct 30 15:13:44.580513 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:44.580476    5064 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9013-987220\" is forbidden: User \"system:node:ip-10-0-138-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-143.ec2.internal' and this object" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" pod="opendatahub/test-curl-9013-987220"
Oct 30 15:13:44.726400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.726365    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" path="/var/lib/kubelet/pods/7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2/volumes"
Oct 30 15:13:44.979663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.979612    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:13:44.980747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:44.980712    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" podUID="630c7525-1a95-4288-bf16-52750f26e351" containerName="maas-api" containerID="cri-o://670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022" gracePeriod=30
Oct 30 15:13:45.577719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.577687    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:13:45.670397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.670361    5064 generic.go:356] "Generic (PLEG): container finished" podID="630c7525-1a95-4288-bf16-52750f26e351" containerID="670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022" exitCode=0
Oct 30 15:13:45.670818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.670794    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" event={"ID":"630c7525-1a95-4288-bf16-52750f26e351","Type":"ContainerDied","Data":"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"}
Oct 30 15:13:45.671012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.670991    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz" event={"ID":"630c7525-1a95-4288-bf16-52750f26e351","Type":"ContainerDied","Data":"5cee3e0c15513a360f5dfea56a58278ade7f72eb4b8e66ea2759bc42dfcf1a3b"}
Oct 30 15:13:45.671243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.671182    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"
Oct 30 15:13:45.671840 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.671197    5064 scope.go:122] "RemoveContainer" containerID="670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"
Oct 30 15:13:45.695763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.695739    5064 scope.go:122] "RemoveContainer" containerID="670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"
Oct 30 15:13:45.696338 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:45.696315    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022\": container with ID starting with 670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022 not found: ID does not exist" containerID="670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"
Oct 30 15:13:45.696493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.696468    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022"} err="failed to get container status \"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022\": rpc error: code = NotFound desc = could not find container \"670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022\": container with ID starting with 670d38e2a97eedf20b43d3faa89e4f03a006fdf0a31c6e3b24f86aa03669d022 not found: ID does not exist"
Oct 30 15:13:45.730652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.730591    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\") pod \"630c7525-1a95-4288-bf16-52750f26e351\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") "
Oct 30 15:13:45.750792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.741174    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\") pod \"630c7525-1a95-4288-bf16-52750f26e351\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") "
Oct 30 15:13:45.751116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.751086    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\") pod \"630c7525-1a95-4288-bf16-52750f26e351\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") "
Oct 30 15:13:45.759863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.751382    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\" (UniqueName: \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\") pod \"630c7525-1a95-4288-bf16-52750f26e351\" (UID: \"630c7525-1a95-4288-bf16-52750f26e351\") "
Oct 30 15:13:45.881418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.876870    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs" pod "630c7525-1a95-4288-bf16-52750f26e351" (UID: "630c7525-1a95-4288-bf16-52750f26e351"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:13:45.881418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.877467    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca" pod "630c7525-1a95-4288-bf16-52750f26e351" (UID: "630c7525-1a95-4288-bf16-52750f26e351"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:13:45.933103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.902357    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp" pod "630c7525-1a95-4288-bf16-52750f26e351" (UID: "630c7525-1a95-4288-bf16-52750f26e351"). InnerVolumeSpecName "kube-api-access-jgdrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:13:45.938362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.938323    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgdrp\" (UniqueName: \"kubernetes.io/projected/630c7525-1a95-4288-bf16-52750f26e351-kube-api-access-jgdrp\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:13:45.938582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.938563    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:13:45.938672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.938661    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630c7525-1a95-4288-bf16-52750f26e351-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:13:45.977082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:45.977039    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls" pod "630c7525-1a95-4288-bf16-52750f26e351" (UID: "630c7525-1a95-4288-bf16-52750f26e351"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:13:46.044142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.044100    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630c7525-1a95-4288-bf16-52750f26e351-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:13:46.322702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.322668    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:13:46.335867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.335792    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4204df-f96498fbd-bnskz"]
Oct 30 15:13:46.730033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.729995    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630c7525-1a95-4288-bf16-52750f26e351" path="/var/lib/kubelet/pods/630c7525-1a95-4288-bf16-52750f26e351/volumes"
Oct 30 15:13:46.921816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.921779    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:13:46.923150 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923121    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" containerName="test-curl-9013-987220"
Oct 30 15:13:46.923364 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923339    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" containerName="test-curl-9013-987220"
Oct 30 15:13:46.923486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923384    5064 cpu_manager.go:414] "containerMap: removing container" podUID="630c7525-1a95-4288-bf16-52750f26e351" containerName="maas-api"
Oct 30 15:13:46.923486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923393    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630c7525-1a95-4288-bf16-52750f26e351" containerName="maas-api"
Oct 30 15:13:46.923622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923606    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa2543a-c4c0-48b2-8b9d-2de285a9c2b2" containerName="test-curl-9013-987220"
Oct 30 15:13:46.923684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.923627    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="630c7525-1a95-4288-bf16-52750f26e351" containerName="maas-api"
Oct 30 15:13:46.927583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.927561    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:46.946641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.946610    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:13:46.955037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.954989    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24fff1\"/\"test-model-24fff1-kserve-self-signed-certs\""
Oct 30 15:13:46.955195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.955072    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24fff1\"/\"default-dockercfg-4f47g\""
Oct 30 15:13:46.962025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.962000    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24fff1\"/\"openshift-service-ca.crt\""
Oct 30 15:13:46.962191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:46.962088    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24fff1\"/\"kube-root-ca.crt\""
Oct 30 15:13:47.061897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.061806    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttts\" (UniqueName: \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.061897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.061871    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.062307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.061900    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.062307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.062023    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.062307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.062216    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.062307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.062271    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.166257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.166213    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.183558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.183523    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.184557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.184531    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.205595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.205558    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.206431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.206405    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.211217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.211188    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttts\" (UniqueName: \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.211995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.189762    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.212191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.210916    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.212344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.184133    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.228962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.228817    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.229599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.229567    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.281864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.280604    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttts\" (UniqueName: \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\") pod \"test-model-24fff1-kserve-77fd74bd97-qmgm6\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") " pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.544341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.544297    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:47.856436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:47.856404    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:13:47.856645 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:13:47.856611    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41354d05_6baf_43cd_8daf_60f03d07d59a.slice/crio-ddcf803e10df63c9182dd6e92464dd19fe683703f29ed06ed5d1e7689dd1765d WatchSource:0}: Error finding container ddcf803e10df63c9182dd6e92464dd19fe683703f29ed06ed5d1e7689dd1765d: Status 404 returned error can't find the container with id ddcf803e10df63c9182dd6e92464dd19fe683703f29ed06ed5d1e7689dd1765d
Oct 30 15:13:48.701120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:48.701076    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" event={"ID":"41354d05-6baf-43cd-8daf-60f03d07d59a","Type":"ContainerStarted","Data":"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"}
Oct 30 15:13:48.701698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:48.701132    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" event={"ID":"41354d05-6baf-43cd-8daf-60f03d07d59a","Type":"ContainerStarted","Data":"ddcf803e10df63c9182dd6e92464dd19fe683703f29ed06ed5d1e7689dd1765d"}
Oct 30 15:13:48.750811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:48.750761    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" podStartSLOduration=2.7507474690000002 podStartE2EDuration="2.750747469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:48.745311707 +0000 UTC m=+1578.187202426" watchObservedRunningTime="2026-10-30 15:13:48.750747469 +0000 UTC m=+1578.192638186"
Oct 30 15:13:49.544820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:49.544781    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:50.092883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.092849    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:13:50.097508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.097489    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.125890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.118410    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:13:50.125890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.125124    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-751ab3-serving-cert\""
Oct 30 15:13:50.182100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.182053    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.182309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.182213    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.182309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.182286    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtm8\" (UniqueName: \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.182474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.182454    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.283778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.283736    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtm8\" (UniqueName: \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.283991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.283943    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.284103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.284088    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.284214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.284200    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.284823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.284796    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.286813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.286775    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.286961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.286784    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.313327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.313293    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtm8\" (UniqueName: \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\") pod \"maas-api-e2e-api-a-751ab3-84884646b9-2dndm\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.426870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.422238    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:50.921997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:50.915237    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:13:51.856086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:51.854734    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" event={"ID":"c51166b0-5af4-445b-a420-470b3bf24090","Type":"ContainerStarted","Data":"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"}
Oct 30 15:13:51.856086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:51.854805    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" event={"ID":"c51166b0-5af4-445b-a420-470b3bf24090","Type":"ContainerStarted","Data":"4690b09ee6c60d2aa41e03842227efd56929a3343b1ec5c13dcab9292940e405"}
Oct 30 15:13:51.856086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:51.855957    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:52.016142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.016041    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" podStartSLOduration=1.49482111 podStartE2EDuration="2.016016154s" totalImagesPullingTime="521.195044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:51.923087946 +0000 UTC m=+1581.364978679" watchObservedRunningTime="2026-10-30 15:13:52.016016154 +0000 UTC m=+1581.457906917"
Oct 30 15:13:52.446671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.446634    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:13:52.451778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.451748    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.474311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.474278    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:13:52.478183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.478156    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c69f40-serving-cert\""
Oct 30 15:13:52.506367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.506330    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.507349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.507321    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.507615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.507596    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvn2\" (UniqueName: \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.508118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.508090    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.609445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.609406    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvn2\" (UniqueName: \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.609642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.609613    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.609819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.609801    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.609943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.609928    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.610083 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:52.610060    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-serving-cert: secret "maas-api-e2e-api-b-c69f40-serving-cert" not found
Oct 30 15:13:52.610148 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:52.610127    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls podName:ff523d8c-f1e6-4e98-89eb-4f1ccb706dab nodeName:}" failed. No retries permitted until 2026-10-30 15:13:53.110112421 +0000 UTC m=+1582.552003133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls") pod "maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" (UID: "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab") : secret "maas-api-e2e-api-b-c69f40-serving-cert" not found
Oct 30 15:13:52.610557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.610535    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.612368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.612347    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:52.658849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:52.651608    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvn2\" (UniqueName: \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:53.122347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:53.122288    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:53.124985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:53.124957    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") pod \"maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:53.382936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:53.382819    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:53.718412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:53.717973    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:13:53.978575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:53.978479    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" event={"ID":"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab","Type":"ContainerStarted","Data":"7db1e34ebf5f57d0405c2fd8db0af0fb0138535918aeb03cc1c9e3339e599f13"}
Oct 30 15:13:54.991632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:54.990651    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" event={"ID":"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab","Type":"ContainerStarted","Data":"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"}
Oct 30 15:13:54.991632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:54.991631    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:13:55.064902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:55.064775    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" podStartSLOduration=2.505302602 podStartE2EDuration="3.064752908s" totalImagesPullingTime="559.450306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:55.055248858 +0000 UTC m=+1584.497139574" watchObservedRunningTime="2026-10-30 15:13:55.064752908 +0000 UTC m=+1584.506643625"
Oct 30 15:13:57.544811 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:57.544761    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:57.560201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:57.560169    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:58.031119 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:58.031090    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:13:58.984083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:13:58.984053    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:13:59.632922 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:13:59.632878    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:14:01.009442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:01.009412    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:14:22.259375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:22.259340    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:14:22.259928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:22.259553    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" podUID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" containerName="maas-api" containerID="cri-o://a384aea214745189748e75709fa7c4367db3c71ce81061480f4dc5f56390cbf1" gracePeriod=30
Oct 30 15:14:22.972044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:22.971517    5064 generic.go:356] "Generic (PLEG): container finished" podID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" containerID="a384aea214745189748e75709fa7c4367db3c71ce81061480f4dc5f56390cbf1" exitCode=0
Oct 30 15:14:22.988882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:22.986922    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" event={"ID":"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb","Type":"ContainerDied","Data":"a384aea214745189748e75709fa7c4367db3c71ce81061480f4dc5f56390cbf1"}
Oct 30 15:14:23.074860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.074811    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.117526    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\") pod \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") "
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.118050    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca" pod "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" (UID: "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.118257    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\") pod \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") "
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.118511    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\" (UniqueName: \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\") pod \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") "
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.118553    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\") pod \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\" (UID: \"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb\") "
Oct 30 15:14:23.121902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.119487    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:23.144858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.142584    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs" pod "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" (UID: "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:14:23.144858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.142721    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls" pod "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" (UID: "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:14:23.144858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.143126    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm" pod "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" (UID: "238273c2-e6a7-43a2-b5c1-2eb2d3d378bb"). InnerVolumeSpecName "kube-api-access-g46fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:23.222176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.222132    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:23.222176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.222177    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:23.222395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:23.222193    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46fm\" (UniqueName: \"kubernetes.io/projected/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb-kube-api-access-g46fm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:24.024537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.024506    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"
Oct 30 15:14:24.025059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.024493    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv" event={"ID":"238273c2-e6a7-43a2-b5c1-2eb2d3d378bb","Type":"ContainerDied","Data":"0923a5500e5f9594dbb0b631ad50be8a7d8bfc949a4cde591e287d60399dd69e"}
Oct 30 15:14:24.025059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.024657    5064 scope.go:122] "RemoveContainer" containerID="a384aea214745189748e75709fa7c4367db3c71ce81061480f4dc5f56390cbf1"
Oct 30 15:14:24.170014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.169979    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:14:24.183650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.183616    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-wk6fv"]
Oct 30 15:14:24.738314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:24.738275    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" path="/var/lib/kubelet/pods/238273c2-e6a7-43a2-b5c1-2eb2d3d378bb/volumes"
Oct 30 15:14:28.827577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:28.827534    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:14:29.634982 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:14:29.634932    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:14:34.246963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.246927    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:14:34.247745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.247728    5064 cpu_manager.go:414] "containerMap: removing container" podUID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" containerName="maas-api"
Oct 30 15:14:34.247745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.247746    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" containerName="maas-api"
Oct 30 15:14:34.247930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.247919    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="238273c2-e6a7-43a2-b5c1-2eb2d3d378bb" containerName="maas-api"
Oct 30 15:14:34.253539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.253518    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.262657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.262625    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.262761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.262688    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.262892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.262867    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.262969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.262953    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.263093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.263069    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.263163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.263115    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.263229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.263206    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.263348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.263331    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8277\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.263404 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.263392    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.278458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.278426    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:14:34.364741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.364706    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.364944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.364750    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.364944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.364796    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.364950    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8277\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365014    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365177 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365149    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365201    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365201    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365318    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365391    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365445    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365518 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365499    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365525    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.365569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.365554    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.367713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.367691    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.367974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.367959    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.390331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.390303    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.390464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.390431    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8277\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.569798 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.569711    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:34.902074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.902033    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:14:34.903254 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:14:34.903226    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e89feb7_17bd_48fd_b28d_f2f92535a863.slice/crio-84a1bb3503abfc5334e6fd098d6841cd3a2fb693f82fe759c6cf35dda4d5a25e WatchSource:0}: Error finding container 84a1bb3503abfc5334e6fd098d6841cd3a2fb693f82fe759c6cf35dda4d5a25e: Status 404 returned error can't find the container with id 84a1bb3503abfc5334e6fd098d6841cd3a2fb693f82fe759c6cf35dda4d5a25e
Oct 30 15:14:34.906170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.906137    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:14:34.906242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.906222    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:14:34.906291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:34.906252    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:14:35.125022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:35.124985    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" event={"ID":"7e89feb7-17bd-48fd-b28d-f2f92535a863","Type":"ContainerStarted","Data":"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"}
Oct 30 15:14:35.125228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:35.125048    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" event={"ID":"7e89feb7-17bd-48fd-b28d-f2f92535a863","Type":"ContainerStarted","Data":"84a1bb3503abfc5334e6fd098d6841cd3a2fb693f82fe759c6cf35dda4d5a25e"}
Oct 30 15:14:35.570845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:35.570800    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:35.572920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:35.572892    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:35.573011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:35.572950    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:36.570503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:36.570464    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:36.570687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:36.570525    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:37.570478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:37.570435    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:37.570931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:37.570502    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:38.570207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:38.570168    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:38.570413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:38.570236    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:39.570303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:39.570267    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:39.570720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:39.570324    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:40.575410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:40.575349    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:40.576096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:40.576045    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:41.170985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.170926    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podStartSLOduration=7.17090987 podStartE2EDuration="7.17090987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:35.210270462 +0000 UTC m=+1624.652161179" watchObservedRunningTime="2026-10-30 15:14:41.17090987 +0000 UTC m=+1630.612800586"
Oct 30 15:14:41.174981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.174955    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:14:41.175178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.175148    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy" containerID="cri-o://a3513121822819c9e4c74a1d6519bdf87553b554f1ce38cf819384aa589ce543" gracePeriod=30
Oct 30 15:14:41.189416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.189385    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:14:41.189587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.189568    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy" containerID="cri-o://e3034fad677d9e921467512bfdd580457c402944b2f0dfb6e47daa76cf59232b" gracePeriod=30
Oct 30 15:14:41.571141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.571048    5064 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 15:14:41.571141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:41.571122    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 15:14:42.016936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:42.016891    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:14:42.017506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:42.017147    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy" containerID="cri-o://df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd" gracePeriod=30
Oct 30 15:14:42.575342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:42.575308    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:42.575798 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:42.575783    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:42.576461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:42.576442    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:14:43.857474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:43.857427    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerName="main" containerID="cri-o://c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072" gracePeriod=45
Oct 30 15:14:44.154571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.154544    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:14:44.218527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.218487    5064 generic.go:356] "Generic (PLEG): container finished" podID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerID="c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072" exitCode=0
Oct 30 15:14:44.219731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.219706    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"
Oct 30 15:14:44.220133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.220109    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" event={"ID":"97927525-fb17-4e9c-80bd-8d2c47d0dc04","Type":"ContainerDied","Data":"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"}
Oct 30 15:14:44.220248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.220155    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" event={"ID":"97927525-fb17-4e9c-80bd-8d2c47d0dc04","Type":"ContainerDied","Data":"8a7847cea0a4e23da69605499e587ef5aa3fc786f58b420c3fe8cec863de5b51"}
Oct 30 15:14:44.220248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.220179    5064 scope.go:122] "RemoveContainer" containerID="c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"
Oct 30 15:14:44.235493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.235465    5064 scope.go:122] "RemoveContainer" containerID="c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"
Oct 30 15:14:44.235911 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:14:44.235890    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072\": container with ID starting with c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072 not found: ID does not exist" containerID="c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"
Oct 30 15:14:44.236002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.235921    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072"} err="failed to get container status \"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072\": rpc error: code = NotFound desc = could not find container \"c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072\": container with ID starting with c649d36c5404fcb0d90a2f91144015a49c07ea7498605f08032a582684e9c072 not found: ID does not exist"
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.334844    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.334957    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.335011    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.335053    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\" (UniqueName: \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.335112    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.335496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.335141    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\") pod \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\" (UID: \"97927525-fb17-4e9c-80bd-8d2c47d0dc04\") "
Oct 30 15:14:44.336284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.335739    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:44.336284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.336046    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:44.336284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.336279    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.336447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.336296    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.336508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.336444    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:44.338164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.338136    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:14:44.338410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.338375    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:44.338521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.338499    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck" pod "97927525-fb17-4e9c-80bd-8d2c47d0dc04" (UID: "97927525-fb17-4e9c-80bd-8d2c47d0dc04"). InnerVolumeSpecName "kube-api-access-492ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:44.438611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.438521    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.438611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.438555    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/97927525-fb17-4e9c-80bd-8d2c47d0dc04-kube-api-access-492ck\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.438611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.438566    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97927525-fb17-4e9c-80bd-8d2c47d0dc04-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.438611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.438577    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97927525-fb17-4e9c-80bd-8d2c47d0dc04-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:44.573900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.573863    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:14:44.578218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.578191    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5"]
Oct 30 15:14:44.729818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:44.729730    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" path="/var/lib/kubelet/pods/97927525-fb17-4e9c-80bd-8d2c47d0dc04/volumes"
Oct 30 15:14:45.083705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:45.083604    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f7acff45/e2e-lifecycle-model-f7acff45-kserve-7b8c99bb69-x2gb5" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:14:46.321112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.321078    5064 generic.go:356] "Generic (PLEG): container finished" podID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerID="e3034fad677d9e921467512bfdd580457c402944b2f0dfb6e47daa76cf59232b" exitCode=0
Oct 30 15:14:46.321705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.321356    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" event={"ID":"41932f6a-cb4d-4856-8d4b-39b49adb3bba","Type":"ContainerDied","Data":"e3034fad677d9e921467512bfdd580457c402944b2f0dfb6e47daa76cf59232b"}
Oct 30 15:14:46.323820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.323790    5064 generic.go:356] "Generic (PLEG): container finished" podID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerID="a3513121822819c9e4c74a1d6519bdf87553b554f1ce38cf819384aa589ce543" exitCode=0
Oct 30 15:14:46.323993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.323891    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" event={"ID":"80fa1e63-f074-4f35-a83f-80896ce4dea2","Type":"ContainerDied","Data":"a3513121822819c9e4c74a1d6519bdf87553b554f1ce38cf819384aa589ce543"}
Oct 30 15:14:46.618856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.614392    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:14:46.729434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.729397    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:14:46.734195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734159    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734386    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734526    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734600    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734661    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734703    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734755    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734792    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.734981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.734900    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\") pod \"80fa1e63-f074-4f35-a83f-80896ce4dea2\" (UID: \"80fa1e63-f074-4f35-a83f-80896ce4dea2\") "
Oct 30 15:14:46.736102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.736051    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:14:46.736666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.736627    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.736789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.736672    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.737288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.737260    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.737512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.737489    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.740187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.740166    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:46.743265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.743220    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:14:46.743367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.743301    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "kube-api-access-6g69g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:46.744457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.744432    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy" pod "80fa1e63-f074-4f35-a83f-80896ce4dea2" (UID: "80fa1e63-f074-4f35-a83f-80896ce4dea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.837198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837166    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837271    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837313    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837351    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837442    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837509    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837630    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837651    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.837712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837665    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.837870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837751    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.837956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837869    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\") pod \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\" (UID: \"41932f6a-cb4d-4856-8d4b-39b49adb3bba\") "
Oct 30 15:14:46.838035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.837983    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.838035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838010    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.838477 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838449    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:14:46.838885 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838860    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g69g\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-kube-api-access-6g69g\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.838975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838889    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.838975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838906    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.838975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838920    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.838975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838949    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.838975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838971    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.838987    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839000    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839014    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839028    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839043    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa1e63-f074-4f35-a83f-80896ce4dea2-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839057    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa1e63-f074-4f35-a83f-80896ce4dea2-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839070    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa1e63-f074-4f35-a83f-80896ce4dea2-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.839251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.839086    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.840268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.840243    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "kube-api-access-m2pjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:46.840561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.840536    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:46.840890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.840869    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:46.841315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.841289    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo" pod "41932f6a-cb4d-4856-8d4b-39b49adb3bba" (UID: "41932f6a-cb4d-4856-8d4b-39b49adb3bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:14:46.941261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.941224    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.941261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.941257    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.941261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.941267    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41932f6a-cb4d-4856-8d4b-39b49adb3bba-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:46.941509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:46.941279    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2pjs\" (UniqueName: \"kubernetes.io/projected/41932f6a-cb4d-4856-8d4b-39b49adb3bba-kube-api-access-m2pjs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.324612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.324584    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:14:47.349986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.349954    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"
Oct 30 15:14:47.349986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.349982    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn" event={"ID":"41932f6a-cb4d-4856-8d4b-39b49adb3bba","Type":"ContainerDied","Data":"47341813992d1d1f616527a7bd6666083afbc8981ff9f702c99c114da0452579"}
Oct 30 15:14:47.350213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.350047    5064 scope.go:122] "RemoveContainer" containerID="e3034fad677d9e921467512bfdd580457c402944b2f0dfb6e47daa76cf59232b"
Oct 30 15:14:47.352022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.351993    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"
Oct 30 15:14:47.352156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.352045    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp" event={"ID":"80fa1e63-f074-4f35-a83f-80896ce4dea2","Type":"ContainerDied","Data":"9fb78d33fff02436a912303e73c6f095481737bdb081fefa463ea81cc7ded915"}
Oct 30 15:14:47.353679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.353655    5064 generic.go:356] "Generic (PLEG): container finished" podID="15983fe3-07d4-440d-9447-b1122fab9c45" containerID="df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd" exitCode=0
Oct 30 15:14:47.353774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.353754    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" event={"ID":"15983fe3-07d4-440d-9447-b1122fab9c45","Type":"ContainerDied","Data":"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"}
Oct 30 15:14:47.353815 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.353792    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc" event={"ID":"15983fe3-07d4-440d-9447-b1122fab9c45","Type":"ContainerDied","Data":"6a60e8bf71ebb53e0da0926ff21b936ef265da342287ada013e4ed8c1b472965"}
Oct 30 15:14:47.353883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.353872    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"
Oct 30 15:14:47.367123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.367099    5064 scope.go:122] "RemoveContainer" containerID="a3513121822819c9e4c74a1d6519bdf87553b554f1ce38cf819384aa589ce543"
Oct 30 15:14:47.392239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.392211    5064 scope.go:122] "RemoveContainer" containerID="df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"
Oct 30 15:14:47.406479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.406452    5064 scope.go:122] "RemoveContainer" containerID="df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"
Oct 30 15:14:47.407213 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:14:47.407146    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd\": container with ID starting with df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd not found: ID does not exist" containerID="df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"
Oct 30 15:14:47.407213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.407185    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd"} err="failed to get container status \"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd\": rpc error: code = NotFound desc = could not find container \"df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd\": container with ID starting with df3b3b35a7c52416810eb16eb693b6bf1cd52af5eefee1df2e151d8f0ccccebd not found: ID does not exist"
Oct 30 15:14:47.450511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450482    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450519    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450546    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450585    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450621    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450801    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.450947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.450822    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:47.451060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451016    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.451121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451063    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.451121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451093    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:47.451323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451301    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:47.451454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451335    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\") pod \"15983fe3-07d4-440d-9447-b1122fab9c45\" (UID: \"15983fe3-07d4-440d-9447-b1122fab9c45\") "
Oct 30 15:14:47.451526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.451464    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:47.452342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.452314    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.452428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.452348    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.452428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.452363    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.452428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.452377    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.453420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.453108    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:14:47.453701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.453673    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:14:47.454365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.454320    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "kube-api-access-2pzcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:47.454611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.454588    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:14:47.455529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.455506    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:14:47.456983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.456961    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy" pod "15983fe3-07d4-440d-9447-b1122fab9c45" (UID: "15983fe3-07d4-440d-9447-b1122fab9c45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:14:47.457676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.457656    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2f7wn"]
Oct 30 15:14:47.490285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.490249    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:14:47.497892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.497856    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-q7hkp"]
Oct 30 15:14:47.554538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.554450    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15983fe3-07d4-440d-9447-b1122fab9c45-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.554538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.554476    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.554538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.554486    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15983fe3-07d4-440d-9447-b1122fab9c45-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.554538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.554495    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15983fe3-07d4-440d-9447-b1122fab9c45-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.554538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.554507    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzcx\" (UniqueName: \"kubernetes.io/projected/15983fe3-07d4-440d-9447-b1122fab9c45-kube-api-access-2pzcx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:14:47.841915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.838884    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:14:47.880858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:47.873422    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qkmnc"]
Oct 30 15:14:48.726690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:48.726652    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" path="/var/lib/kubelet/pods/15983fe3-07d4-440d-9447-b1122fab9c45/volumes"
Oct 30 15:14:48.727229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:48.727211    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" path="/var/lib/kubelet/pods/41932f6a-cb4d-4856-8d4b-39b49adb3bba/volumes"
Oct 30 15:14:48.727690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:48.727677    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" path="/var/lib/kubelet/pods/80fa1e63-f074-4f35-a83f-80896ce4dea2/volumes"
Oct 30 15:14:56.879660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.879618    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:14:56.880491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880472    5064 cpu_manager.go:414] "containerMap: removing container" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerName="main"
Oct 30 15:14:56.880491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880492    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerName="main"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880536    5064 cpu_manager.go:414] "containerMap: removing container" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880541    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880558    5064 cpu_manager.go:414] "containerMap: removing container" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880563    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880577    5064 cpu_manager.go:414] "containerMap: removing container" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy"
Oct 30 15:14:56.880648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880581    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy"
Oct 30 15:14:56.880992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880686    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="41932f6a-cb4d-4856-8d4b-39b49adb3bba" containerName="istio-proxy"
Oct 30 15:14:56.880992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880700    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="97927525-fb17-4e9c-80bd-8d2c47d0dc04" containerName="main"
Oct 30 15:14:56.880992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880713    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="15983fe3-07d4-440d-9447-b1122fab9c45" containerName="istio-proxy"
Oct 30 15:14:56.880992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.880721    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fa1e63-f074-4f35-a83f-80896ce4dea2" containerName="istio-proxy"
Oct 30 15:14:56.885842 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.885802    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:56.927884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.917509    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:14:56.942438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.942403    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a65941-serving-cert\""
Oct 30 15:14:56.993033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.992989    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:14:56.993274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:56.993081    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:14:57.035821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.035784    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.036022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.035909    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.036022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.035948    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.036102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.036094    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.137649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.137553    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.137867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.137664    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.137867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.137697    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.138017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.137867    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.138533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.138500    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.140383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.140359    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.140579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.140555    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.163485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.163447    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\") pod \"maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.271445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.271374    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:57.669879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.660219    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:14:57.771055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:57.771020    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" event={"ID":"ced65a45-9933-453e-a2a1-846a3936d141","Type":"ContainerStarted","Data":"0ed2e3ac36e61dde17c524cc97f27150eb2c1d7b4f1849e0c4ebb180f13ed24a"}
Oct 30 15:14:58.786573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:58.786459    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" event={"ID":"ced65a45-9933-453e-a2a1-846a3936d141","Type":"ContainerStarted","Data":"55d2f2afc4acf4f40ccfa415438ff951165b8104c1cb27c4003d95e77ed3c9ac"}
Oct 30 15:14:58.787114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:58.786782    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:14:58.867010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:14:58.866945    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" podStartSLOduration=2.35877131 podStartE2EDuration="2.866924602s" totalImagesPullingTime="508.153292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:58.864217791 +0000 UTC m=+1648.306108519" watchObservedRunningTime="2026-10-30 15:14:58.866924602 +0000 UTC m=+1648.308815320"
Oct 30 15:14:59.633699 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:14:59.633641    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:15:00.805886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.805818    5064 generic.go:356] "Generic (PLEG): container finished" podID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerID="0df976cd18230e7e3367b2fb03c8d5ece89c9136ce2194d30eac6d8b8bc9e83e" exitCode=0
Oct 30 15:15:00.806341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.805902    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerDied","Data":"0df976cd18230e7e3367b2fb03c8d5ece89c9136ce2194d30eac6d8b8bc9e83e"}
Oct 30 15:15:00.806551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.806532    5064 scope.go:122] "RemoveContainer" containerID="0df976cd18230e7e3367b2fb03c8d5ece89c9136ce2194d30eac6d8b8bc9e83e"
Oct 30 15:15:00.808893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.808863    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:00.808980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.808945    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:00.809017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:00.808981    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:01.846329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:01.846292    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerStarted","Data":"63e55d0724fc368296b5966be3386322746a51d262d5af517eb70be7ca5c6152"}
Oct 30 15:15:02.490579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:02.490547    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:15:02.490787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:02.490761    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 15:15:02.490908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:02.490816    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 15:15:03.495424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:03.495350    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 15:15:03.496374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:03.495480    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 15:15:04.490302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:04.490266    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 15:15:04.490488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:04.490348    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 15:15:04.802001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:04.801926    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:15:05.490266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:05.490229    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 15:15:05.490443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:05.490290    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 15:15:06.490662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:06.490629    5064 patch_prober.go:29] interesting pod/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 15:15:06.491084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:06.490698    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 15:15:07.495797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:07.495765    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:15:07.496326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:07.496196    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:15:07.496816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:07.496795    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:15:15.313664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:15.313571    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:15:15.314238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:15.313824    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy" containerID="cri-o://0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735" gracePeriod=30
Oct 30 15:15:15.320926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:15.320888    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:15:15.321153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:15.321128    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy" containerID="cri-o://90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1" gracePeriod=30
Oct 30 15:15:20.655333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.655299    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:15:20.734955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.734917    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.734990    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735177    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:20.735216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735190    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735286    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735327    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735332    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:20.735395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735362    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735403    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735459    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735663    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\") pod \"7e89feb7-17bd-48fd-b28d-f2f92535a863\" (UID: \"7e89feb7-17bd-48fd-b28d-f2f92535a863\") "
Oct 30 15:15:20.735979 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.735953    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:15:20.736493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736455    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:20.736676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736653    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.736765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736683    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e89feb7-17bd-48fd-b28d-f2f92535a863-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.736765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736696    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.736765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736710    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.736765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.736724    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:20.738361 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.738332    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "kube-api-access-t8277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:20.739008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.738986    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:15:20.739160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.739137    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:20.739374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.739355    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token" pod "7e89feb7-17bd-48fd-b28d-f2f92535a863" (UID: "7e89feb7-17bd-48fd-b28d-f2f92535a863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:20.837723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.837688    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8277\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-kube-api-access-t8277\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.837723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.837717    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.837723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.837727    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.837990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.837735    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e89feb7-17bd-48fd-b28d-f2f92535a863-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:20.837990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:20.837744    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e89feb7-17bd-48fd-b28d-f2f92535a863-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.031641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.031613    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:15:21.143690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.143650    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.143929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.143721    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.143929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.143802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.143965    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144059    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:21.144112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144100    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144169    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:21.144220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144183    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144223    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144296    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144323    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:21.144427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144404    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\") pod \"75c8c585-4cea-4bed-95fd-408cf3362a2c\" (UID: \"75c8c585-4cea-4bed-95fd-408cf3362a2c\") "
Oct 30 15:15:21.144663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144635    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.144870    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.145184    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.145215    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.145230    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.145243    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c8c585-4cea-4bed-95fd-408cf3362a2c-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.145291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.145255    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.146552 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.146529    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:15:21.147004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.146981    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:21.147096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.147017    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "kube-api-access-mkrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:21.147957 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.147936    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy" pod "75c8c585-4cea-4bed-95fd-408cf3362a2c" (UID: "75c8c585-4cea-4bed-95fd-408cf3362a2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:21.151247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.151223    5064 generic.go:356] "Generic (PLEG): container finished" podID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerID="0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735" exitCode=0
Oct 30 15:15:21.151349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.151267    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"
Oct 30 15:15:21.151349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.151313    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" event={"ID":"7e89feb7-17bd-48fd-b28d-f2f92535a863","Type":"ContainerDied","Data":"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"}
Oct 30 15:15:21.151432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.151370    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml" event={"ID":"7e89feb7-17bd-48fd-b28d-f2f92535a863","Type":"ContainerDied","Data":"84a1bb3503abfc5334e6fd098d6841cd3a2fb693f82fe759c6cf35dda4d5a25e"}
Oct 30 15:15:21.151432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.151401    5064 scope.go:122] "RemoveContainer" containerID="0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"
Oct 30 15:15:21.153504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.153481    5064 generic.go:356] "Generic (PLEG): container finished" podID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerID="90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1" exitCode=0
Oct 30 15:15:21.153617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.153535    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"
Oct 30 15:15:21.153683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.153627    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerDied","Data":"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"}
Oct 30 15:15:21.153683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.153659    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22" event={"ID":"75c8c585-4cea-4bed-95fd-408cf3362a2c","Type":"ContainerDied","Data":"efeada85cd486a2a1e50868c927b2edaf8afacc11b0c811dfa8f78e494f43663"}
Oct 30 15:15:21.167129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.167100    5064 scope.go:122] "RemoveContainer" containerID="0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"
Oct 30 15:15:21.168111 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:15:21.168069    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735\": container with ID starting with 0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735 not found: ID does not exist" containerID="0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"
Oct 30 15:15:21.168226 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.168126    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735"} err="failed to get container status \"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735\": rpc error: code = NotFound desc = could not find container \"0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735\": container with ID starting with 0c495eb7a22fb93e71c6badfd6a1a0fc0ee174db9dbebd30a01863c0f6e5e735 not found: ID does not exist"
Oct 30 15:15:21.168226 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.168169    5064 scope.go:122] "RemoveContainer" containerID="90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"
Oct 30 15:15:21.204156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.204112    5064 scope.go:122] "RemoveContainer" containerID="650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"
Oct 30 15:15:21.216886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.216805    5064 scope.go:122] "RemoveContainer" containerID="90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"
Oct 30 15:15:21.217220 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:15:21.217197    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1\": container with ID starting with 90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1 not found: ID does not exist" containerID="90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"
Oct 30 15:15:21.217306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.217236    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1"} err="failed to get container status \"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1\": rpc error: code = NotFound desc = could not find container \"90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1\": container with ID starting with 90a24b63c87c08249cf06cc4972250cadafa8d1b8bec4feb4a0e5ea36326c2a1 not found: ID does not exist"
Oct 30 15:15:21.217306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.217263    5064 scope.go:122] "RemoveContainer" containerID="650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"
Oct 30 15:15:21.217643 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:15:21.217612    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25\": container with ID starting with 650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25 not found: ID does not exist" containerID="650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"
Oct 30 15:15:21.217751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.217648    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25"} err="failed to get container status \"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25\": rpc error: code = NotFound desc = could not find container \"650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25\": container with ID starting with 650c75742f19c893d2e2acf947eab7ddf1f5b02489e1ca13d240abfcc7835b25 not found: ID does not exist"
Oct 30 15:15:21.228514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.228442    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:15:21.245341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.245305    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-29jml"]
Oct 30 15:15:21.246375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.246354    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.246451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.246377    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.246451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.246388    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrbk\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-kube-api-access-mkrbk\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.246451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.246399    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c8c585-4cea-4bed-95fd-408cf3362a2c-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:21.281617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.281580    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:15:21.286266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:21.286230    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-qgb22"]
Oct 30 15:15:22.725868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:22.725809    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" path="/var/lib/kubelet/pods/75c8c585-4cea-4bed-95fd-408cf3362a2c/volumes"
Oct 30 15:15:22.726584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:22.726565    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" path="/var/lib/kubelet/pods/7e89feb7-17bd-48fd-b28d-f2f92535a863/volumes"
Oct 30 15:15:29.633270 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:15:29.633228    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:15:33.635339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.635307    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:15:33.636445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636419    5064 cpu_manager.go:414] "containerMap: removing container" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:33.636445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636445    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:33.636712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636476    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy"
Oct 30 15:15:33.636712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636481    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy"
Oct 30 15:15:33.636712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636647    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:33.636712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.636671    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e89feb7-17bd-48fd-b28d-f2f92535a863" containerName="istio-proxy"
Oct 30 15:15:33.648504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.643408    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.669678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.664940    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:15:33.671890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.671860    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-34218d-openshift-default-dockercfg-wgwq4\""
Oct 30 15:15:33.730470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730427    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.730656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730486    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.730700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730660    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.730743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730723    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.730798 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730755    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.730942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730924    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.731026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.730963    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.731156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.731139    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6pd\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.731260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.731242    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834341    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6pd\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834474    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834637    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834688    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834809    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834880    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834912    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.834999    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.844851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.835034    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.868851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.860568    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.868851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.860942    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.868851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.861253    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.868851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.861395    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.880856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.872651    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.880856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.873561    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.902974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.894241    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.929928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.922388    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.939855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.930680    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6pd\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\") pod \"e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") " pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:33.974852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:33.974136    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:34.515965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.515928    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:15:34.516261 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:15:34.516237    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6991f3f_7592_4998_be53_204e55364fc3.slice/crio-4a5eaa5419e4ce0dd14f06a4a835419f8d77060cbd4bddcacfa63b497cdb5aae WatchSource:0}: Error finding container 4a5eaa5419e4ce0dd14f06a4a835419f8d77060cbd4bddcacfa63b497cdb5aae: Status 404 returned error can't find the container with id 4a5eaa5419e4ce0dd14f06a4a835419f8d77060cbd4bddcacfa63b497cdb5aae
Oct 30 15:15:34.519266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.519228    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:34.519365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.519306    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:34.519365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.519339    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:34.914130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.914096    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:15:34.914929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.914909    5064 cpu_manager.go:414] "containerMap: removing container" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:34.914929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.914926    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:34.915117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.915100    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c8c585-4cea-4bed-95fd-408cf3362a2c" containerName="istio-proxy"
Oct 30 15:15:34.919056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.919033    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:34.941649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.939024    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-496acb-serving-cert\""
Oct 30 15:15:34.945058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.945027    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:15:34.970805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.970750    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:34.971123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.971030    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:34.971218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.971144    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:34.971280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:34.971254    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkln\" (UniqueName: \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.073275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.073230    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkln\" (UniqueName: \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.073480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.073354    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.073642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.073613    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.073767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.073714    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.074489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.074464    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.076370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.076342    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.076506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.076344    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.098727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.098688    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkln\" (UniqueName: \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\") pod \"maas-api-e2e-api-b-496acb-778648cc76-nd8cx\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.239907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.236134    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:35.326504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.326471    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerStarted","Data":"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"}
Oct 30 15:15:35.326732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.326523    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerStarted","Data":"4a5eaa5419e4ce0dd14f06a4a835419f8d77060cbd4bddcacfa63b497cdb5aae"}
Oct 30 15:15:35.391221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.391147    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podStartSLOduration=2.391127738 podStartE2EDuration="2.391127738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:35.377203444 +0000 UTC m=+1684.819094164" watchObservedRunningTime="2026-10-30 15:15:35.391127738 +0000 UTC m=+1684.833018457"
Oct 30 15:15:35.627672 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:15:35.627637    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45f4ed1_abf4_4068_a2d9_264b9b9d8125.slice/crio-e19f52518d71a2440e27d7b6ddf24e3127f321e0c1a59b95f2dadf1f439d27a0 WatchSource:0}: Error finding container e19f52518d71a2440e27d7b6ddf24e3127f321e0c1a59b95f2dadf1f439d27a0: Status 404 returned error can't find the container with id e19f52518d71a2440e27d7b6ddf24e3127f321e0c1a59b95f2dadf1f439d27a0
Oct 30 15:15:35.627824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.627805    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:15:35.629117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.629096    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:15:35.974935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.974805    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:35.981490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:35.981459    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:36.558975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.558905    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" event={"ID":"c45f4ed1-abf4-4068-a2d9-264b9b9d8125","Type":"ContainerStarted","Data":"b897bfca622ab6806be6fa7046fdd73e6a616c3628004fb623324773fc75a123"}
Oct 30 15:15:36.559396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.559319    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:36.559396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.559389    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" event={"ID":"c45f4ed1-abf4-4068-a2d9-264b9b9d8125","Type":"ContainerStarted","Data":"e19f52518d71a2440e27d7b6ddf24e3127f321e0c1a59b95f2dadf1f439d27a0"}
Oct 30 15:15:36.559602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.559573    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:36.561270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.561248    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:15:36.609450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:36.609383    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" podStartSLOduration=2.213262977 podStartE2EDuration="2.609366863s" totalImagesPullingTime="396.103886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:36.605562212 +0000 UTC m=+1686.047452992" watchObservedRunningTime="2026-10-30 15:15:36.609366863 +0000 UTC m=+1686.051257581"
Oct 30 15:15:42.572237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:42.572206    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:15:47.321047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:47.320994    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:15:47.322282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:47.322252    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy" containerID="cri-o://e38b1a15bbcb5b0f93d1ac63e5efd4a41ca3b19ba8c1783655267c7159f94bde" gracePeriod=30
Oct 30 15:15:47.604855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:47.592893    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:15:47.604855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:47.593266    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy" containerID="cri-o://adac9558f9dc8ed29348504d322b23f1758e369b81d64c1578d59e09a04309a9" gracePeriod=30
Oct 30 15:15:52.918679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.918638    5064 generic.go:356] "Generic (PLEG): container finished" podID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerID="adac9558f9dc8ed29348504d322b23f1758e369b81d64c1578d59e09a04309a9" exitCode=0
Oct 30 15:15:52.919136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.918758    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" event={"ID":"b6b28df9-cdce-4ba2-b170-390a0b2d229d","Type":"ContainerDied","Data":"adac9558f9dc8ed29348504d322b23f1758e369b81d64c1578d59e09a04309a9"}
Oct 30 15:15:52.920680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.920652    5064 generic.go:356] "Generic (PLEG): container finished" podID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerID="e38b1a15bbcb5b0f93d1ac63e5efd4a41ca3b19ba8c1783655267c7159f94bde" exitCode=0
Oct 30 15:15:52.920941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.920922    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" event={"ID":"4abee056-d875-4f97-a6a9-c4b9a8269c90","Type":"ContainerDied","Data":"e38b1a15bbcb5b0f93d1ac63e5efd4a41ca3b19ba8c1783655267c7159f94bde"}
Oct 30 15:15:52.921038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.920976    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl" event={"ID":"4abee056-d875-4f97-a6a9-c4b9a8269c90","Type":"ContainerDied","Data":"396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa"}
Oct 30 15:15:52.921038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.921002    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="396c32b51f1615b9b05667eff7d16314f94d5da951b9c52d0eeeefbb34b332fa"
Oct 30 15:15:52.925629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.925607    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:15:52.982687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.981740    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.982687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.981845    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.982687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.981903    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.982687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.981990    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.982687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.982548    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:52.983122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.982907    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:52.984142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.984054    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.984142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.984136    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.984337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.984189    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.984337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.984244    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.984466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.984453    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\") pod \"4abee056-d875-4f97-a6a9-c4b9a8269c90\" (UID: \"4abee056-d875-4f97-a6a9-c4b9a8269c90\") "
Oct 30 15:15:52.985267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.985216    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:52.985267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.985246    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:52.986353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.986085    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:52.986353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.986160    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:52.986353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.986319    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:15:52.986659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.986629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:15:52.992360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.989415    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:52.992360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:52.992037    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "kube-api-access-4qbm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:53.009161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.009123    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy" pod "4abee056-d875-4f97-a6a9-c4b9a8269c90" (UID: "4abee056-d875-4f97-a6a9-c4b9a8269c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.047773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.047740    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088150    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088194    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088210    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qbm7\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-kube-api-access-4qbm7\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088226    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abee056-d875-4f97-a6a9-c4b9a8269c90-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088240    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088258    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.089591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.088270    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abee056-d875-4f97-a6a9-c4b9a8269c90-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.189715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.189617    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.189955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.189740    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.189955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.189762    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.189955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.189792    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.189950    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190065    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.190129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190088    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190091    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.190129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190123    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190177    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190271    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\") pod \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\" (UID: \"b6b28df9-cdce-4ba2-b170-390a0b2d229d\") "
Oct 30 15:15:53.190432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190331    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:15:53.190548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190528    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.190605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.190545    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.191329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.191297    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.191482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.191331    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.191482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.191346    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.191613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.191359    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.191613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.191555    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.192539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.192507    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:15:53.192649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.192613    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:53.192930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.192898    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "kube-api-access-rjsbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:15:53.193024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.193011    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy" pod "b6b28df9-cdce-4ba2-b170-390a0b2d229d" (UID: "b6b28df9-cdce-4ba2-b170-390a0b2d229d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:15:53.293135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.293094    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.293135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.293135    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.293401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.293148    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b28df9-cdce-4ba2-b170-390a0b2d229d-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.293401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.293161    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsbl\" (UniqueName: \"kubernetes.io/projected/b6b28df9-cdce-4ba2-b170-390a0b2d229d-kube-api-access-rjsbl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:15:53.497814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.497714    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:15:53.498961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.498934    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy"
Oct 30 15:15:53.498961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.498959    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy"
Oct 30 15:15:53.499266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.499046    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy"
Oct 30 15:15:53.499266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.499056    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy"
Oct 30 15:15:53.499406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.499274    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" containerName="istio-proxy"
Oct 30 15:15:53.499406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.499295    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" containerName="istio-proxy"
Oct 30 15:15:53.504179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.504157    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.524315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.523821    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-765228-openshift-default-dockercfg-kmvbt\""
Oct 30 15:15:53.531203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.529843    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:15:53.598263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598221    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598292    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsjg\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598386    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598488    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598587    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598635    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598714    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598749    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.598801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.598782    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.722313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.722274    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.724035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.724009    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.724700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.724677    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.725239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.725218    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.725715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.725698    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsjg\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.726795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.726772    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.727429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.727392    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.764648    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.764746    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.725622    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.759500    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsjg\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.759977    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.766025    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.727895    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.766656    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.769899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.722913    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.780865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.773782    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.808861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.808334    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\") pod \"e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") " pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.846639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.844249    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:53.954248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.954215    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"
Oct 30 15:15:53.990853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.960031    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"
Oct 30 15:15:54.024969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:53.969022    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc" event={"ID":"b6b28df9-cdce-4ba2-b170-390a0b2d229d","Type":"ContainerDied","Data":"6f09f902496bf201efe60b7daa2431b3de32908c3ca3eba09369b8d3fd892f5a"}
Oct 30 15:15:54.024969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.019941    5064 scope.go:122] "RemoveContainer" containerID="adac9558f9dc8ed29348504d322b23f1758e369b81d64c1578d59e09a04309a9"
Oct 30 15:15:54.101007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.100001    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:15:54.105049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.105018    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-xmcfc"]
Oct 30 15:15:54.148926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.148869    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:15:54.158430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.158390    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-6kxdl"]
Oct 30 15:15:54.487176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.479365    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:54.487176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.479458    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:54.487176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.479505    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:15:54.574894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.563807    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:15:54.728794 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.728714    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abee056-d875-4f97-a6a9-c4b9a8269c90" path="/var/lib/kubelet/pods/4abee056-d875-4f97-a6a9-c4b9a8269c90/volumes"
Oct 30 15:15:54.729416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.729396    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b28df9-cdce-4ba2-b170-390a0b2d229d" path="/var/lib/kubelet/pods/b6b28df9-cdce-4ba2-b170-390a0b2d229d/volumes"
Oct 30 15:15:54.963442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.963401    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerStarted","Data":"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"}
Oct 30 15:15:54.964306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:54.963467    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerStarted","Data":"2f0289245aa73c350de6d9fb183e18029577ac8bb0d40425612fb79bc3f8f437"}
Oct 30 15:15:55.012964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:55.012853    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podStartSLOduration=2.01281486 podStartE2EDuration="2.01281486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:55.011509752 +0000 UTC m=+1704.453400504" watchObservedRunningTime="2026-10-30 15:15:55.01281486 +0000 UTC m=+1704.454705577"
Oct 30 15:15:55.845643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:55.845602    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:55.852151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:55.852119    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:55.972726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:55.972683    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:55.973874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:15:55.973818    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:15:59.930064 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:15:59.930020    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:16:08.380300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:08.380264    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:16:08.380966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:08.380570    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" podUID="93ea3586-be8b-4246-a2b7-42f3c399883a" containerName="maas-api" containerID="cri-o://81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3" gracePeriod=30
Oct 30 15:16:09.148898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.141436    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:16:09.224553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.224513    5064 generic.go:356] "Generic (PLEG): container finished" podID="93ea3586-be8b-4246-a2b7-42f3c399883a" containerID="81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3" exitCode=0
Oct 30 15:16:09.224739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.224672    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" event={"ID":"93ea3586-be8b-4246-a2b7-42f3c399883a","Type":"ContainerDied","Data":"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"}
Oct 30 15:16:09.224739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.224719    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g" event={"ID":"93ea3586-be8b-4246-a2b7-42f3c399883a","Type":"ContainerDied","Data":"513875e6c6e16a444dd74f70e092dd2b9c892352334bde7cbc6e7ba60c718119"}
Oct 30 15:16:09.224860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.224751    5064 scope.go:122] "RemoveContainer" containerID="81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"
Oct 30 15:16:09.224965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.224947    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"
Oct 30 15:16:09.250026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.249999    5064 scope.go:122] "RemoveContainer" containerID="81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"
Oct 30 15:16:09.250273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.250247    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\") pod \"93ea3586-be8b-4246-a2b7-42f3c399883a\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") "
Oct 30 15:16:09.250371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.250326    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\") pod \"93ea3586-be8b-4246-a2b7-42f3c399883a\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") "
Oct 30 15:16:09.250442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.250380    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\") pod \"93ea3586-be8b-4246-a2b7-42f3c399883a\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") "
Oct 30 15:16:09.250507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.250498    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\") pod \"93ea3586-be8b-4246-a2b7-42f3c399883a\" (UID: \"93ea3586-be8b-4246-a2b7-42f3c399883a\") "
Oct 30 15:16:09.251893 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:09.250669    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3\": container with ID starting with 81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3 not found: ID does not exist" containerID="81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"
Oct 30 15:16:09.252249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.252208    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3"} err="failed to get container status \"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3\": rpc error: code = NotFound desc = could not find container \"81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3\": container with ID starting with 81f7639577ef4fe5c59e278809c2bcf4657fc53235db6183f0550bd2864e6cf3 not found: ID does not exist"
Oct 30 15:16:09.252387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.251248    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca" pod "93ea3586-be8b-4246-a2b7-42f3c399883a" (UID: "93ea3586-be8b-4246-a2b7-42f3c399883a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:16:09.253864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.253798    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs" pod "93ea3586-be8b-4246-a2b7-42f3c399883a" (UID: "93ea3586-be8b-4246-a2b7-42f3c399883a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:16:09.254133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.254108    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls" pod "93ea3586-be8b-4246-a2b7-42f3c399883a" (UID: "93ea3586-be8b-4246-a2b7-42f3c399883a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:16:09.256594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.256571    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q" pod "93ea3586-be8b-4246-a2b7-42f3c399883a" (UID: "93ea3586-be8b-4246-a2b7-42f3c399883a"). InnerVolumeSpecName "kube-api-access-67n9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:09.357942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.356493    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n9q\" (UniqueName: \"kubernetes.io/projected/93ea3586-be8b-4246-a2b7-42f3c399883a-kube-api-access-67n9q\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:09.357942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.356536    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:09.357942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.356552    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea3586-be8b-4246-a2b7-42f3c399883a-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:09.357942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.356569    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ea3586-be8b-4246-a2b7-42f3c399883a-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:09.629855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.628903    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:16:09.651537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:09.648968    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-147bdb-864c69d766-9sh5g"]
Oct 30 15:16:10.725140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:10.725104    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ea3586-be8b-4246-a2b7-42f3c399883a" path="/var/lib/kubelet/pods/93ea3586-be8b-4246-a2b7-42f3c399883a/volumes"
Oct 30 15:16:11.647586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:11.647446    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 15:16:11.648167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:11.648122    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy" containerID="cri-o://a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307" gracePeriod=30
Oct 30 15:16:16.458130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:16.458095    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:16:16.458654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:16.458321    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy" containerID="cri-o://857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b" gracePeriod=30
Oct 30 15:16:17.189178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.189145    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:16:17.266944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.266907    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267180    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267302    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267355    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267428    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267467    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267546    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267600    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.267742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267628    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\") pod \"28085bb2-5d6a-43fe-b036-af6921999478\" (UID: \"28085bb2-5d6a-43fe-b036-af6921999478\") "
Oct 30 15:16:17.268077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.267965    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:16:17.268534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.268500    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28085bb2-5d6a-43fe-b036-af6921999478-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.268739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.268718    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:17.269036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.268958    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:17.269262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.269233    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:17.269748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.269720    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "kube-api-access-vf5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:17.271058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.271034    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:16:17.276057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.275970    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:17.276959 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.276932    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:17.277162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.277138    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token" pod "28085bb2-5d6a-43fe-b036-af6921999478" (UID: "28085bb2-5d6a-43fe-b036-af6921999478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:17.296962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.296900    5064 generic.go:356] "Generic (PLEG): container finished" podID="28085bb2-5d6a-43fe-b036-af6921999478" containerID="a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307" exitCode=0
Oct 30 15:16:17.296962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.296939    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"
Oct 30 15:16:17.298468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.297159    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerDied","Data":"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"}
Oct 30 15:16:17.298468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.297239    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62" event={"ID":"28085bb2-5d6a-43fe-b036-af6921999478","Type":"ContainerDied","Data":"1c20e629274b7e90842f6d6ab9b585434037e0ee36fa17f8bca923e125d49885"}
Oct 30 15:16:17.298468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.297275    5064 scope.go:122] "RemoveContainer" containerID="a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"
Oct 30 15:16:17.311358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.311336    5064 scope.go:122] "RemoveContainer" containerID="fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"
Oct 30 15:16:17.326456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.326431    5064 scope.go:122] "RemoveContainer" containerID="a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"
Oct 30 15:16:17.326959 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:17.326931    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307\": container with ID starting with a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307 not found: ID does not exist" containerID="a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"
Oct 30 15:16:17.327073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.326967    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307"} err="failed to get container status \"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307\": rpc error: code = NotFound desc = could not find container \"a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307\": container with ID starting with a1fb080211ba8bd2344e48a0771b6c4ffeca41060a7f323937e75b671a4a0307 not found: ID does not exist"
Oct 30 15:16:17.327073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.326990    5064 scope.go:122] "RemoveContainer" containerID="fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"
Oct 30 15:16:17.327391 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:17.327349    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59\": container with ID starting with fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59 not found: ID does not exist" containerID="fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"
Oct 30 15:16:17.327492 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.327401    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59"} err="failed to get container status \"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59\": rpc error: code = NotFound desc = could not find container \"fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59\": container with ID starting with fd58520df207f0519a7b1ba34f9d93f5b026d7caaac71e4c3f02167db72c4c59 not found: ID does not exist"
Oct 30 15:16:17.366092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.366068    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 15:16:17.370007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.369977    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28085bb2-5d6a-43fe-b036-af6921999478-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370007    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370021    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370033    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370044    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370054    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28085bb2-5d6a-43fe-b036-af6921999478-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370068    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5fp\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-kube-api-access-vf5fp\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370080    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28085bb2-5d6a-43fe-b036-af6921999478-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:17.370197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:17.370154    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-tvn62"]
Oct 30 15:16:18.725926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:18.725887    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28085bb2-5d6a-43fe-b036-af6921999478" path="/var/lib/kubelet/pods/28085bb2-5d6a-43fe-b036-af6921999478/volumes"
Oct 30 15:16:21.764717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.764691    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:16:21.865554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865519    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.865755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865573    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.865755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865616    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.865902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865778    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.865902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865896    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.866021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.865973    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.866080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.866037    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.866351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.866104    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.866474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.866452    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:21.866807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.866782    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:21.866967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.866857    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.867196 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.867141    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:16:21.867767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.867741    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:21.870526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.870505    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:21.871358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.871223    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:21.871358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.871267    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "kube-api-access-flhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:21.871546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.871408    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:16:21.968040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968007    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\") pod \"c00805fd-dea5-414c-8a93-4a7af0dc70df\" (UID: \"c00805fd-dea5-414c-8a93-4a7af0dc70df\") "
Oct 30 15:16:21.968331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968306    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs" pod "c00805fd-dea5-414c-8a93-4a7af0dc70df" (UID: "c00805fd-dea5-414c-8a93-4a7af0dc70df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:21.968547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968531    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968549    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968558    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968566    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968574    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968582    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968589    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhzf\" (UniqueName: \"kubernetes.io/projected/c00805fd-dea5-414c-8a93-4a7af0dc70df-kube-api-access-flhzf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:21.968774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:21.968599    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00805fd-dea5-414c-8a93-4a7af0dc70df-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:22.378409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.378372    5064 generic.go:356] "Generic (PLEG): container finished" podID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerID="857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b" exitCode=0
Oct 30 15:16:22.378580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.378430    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"
Oct 30 15:16:22.378580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.378452    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerDied","Data":"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"}
Oct 30 15:16:22.378580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.378521    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx" event={"ID":"c00805fd-dea5-414c-8a93-4a7af0dc70df","Type":"ContainerDied","Data":"f07c819a5331df2ad7965870bc511a856dddcbda598876576135bc67abcde71f"}
Oct 30 15:16:22.378697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.378582    5064 scope.go:122] "RemoveContainer" containerID="857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"
Oct 30 15:16:22.390247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.390228    5064 scope.go:122] "RemoveContainer" containerID="871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"
Oct 30 15:16:22.401555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.401534    5064 scope.go:122] "RemoveContainer" containerID="857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"
Oct 30 15:16:22.401895 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:22.401869    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b\": container with ID starting with 857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b not found: ID does not exist" containerID="857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"
Oct 30 15:16:22.401946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.401907    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b"} err="failed to get container status \"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b\": rpc error: code = NotFound desc = could not find container \"857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b\": container with ID starting with 857a2c3fa2bd6d70abc0c04cf9ed32d6052bcd067accbfc35e2dd581e30aef2b not found: ID does not exist"
Oct 30 15:16:22.401946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.401927    5064 scope.go:122] "RemoveContainer" containerID="871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"
Oct 30 15:16:22.402251 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:22.402230    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19\": container with ID starting with 871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19 not found: ID does not exist" containerID="871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"
Oct 30 15:16:22.402345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.402257    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19"} err="failed to get container status \"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19\": rpc error: code = NotFound desc = could not find container \"871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19\": container with ID starting with 871ed01cc7fbf1eaf865091859cce2e091b07017a511409ca00a95dfb0656d19 not found: ID does not exist"
Oct 30 15:16:22.452297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.452261    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:16:22.466244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.466206    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-147bdb-openshift-default-6d95584989-bwkqx"]
Oct 30 15:16:22.726650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:22.726566    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" path="/var/lib/kubelet/pods/c00805fd-dea5-414c-8a93-4a7af0dc70df/volumes"
Oct 30 15:16:24.876008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.875660    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.876867    5064 cpu_manager.go:414] "containerMap: removing container" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.876892    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.876927    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.876959    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877028    5064 cpu_manager.go:414] "containerMap: removing container" podUID="93ea3586-be8b-4246-a2b7-42f3c399883a" containerName="maas-api"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877039    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ea3586-be8b-4246-a2b7-42f3c399883a" containerName="maas-api"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877227    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877241    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877256    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:24.880942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.877277    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ea3586-be8b-4246-a2b7-42f3c399883a" containerName="maas-api"
Oct 30 15:16:24.917736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.916761    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:16:24.917736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.916950    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:24.934903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.934363    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-34218d-serving-cert\""
Oct 30 15:16:24.941298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.939810    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:24.941298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.939951    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:24.941298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.940640    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9wt\" (UniqueName: \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:24.941568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:24.941468    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.043235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.043193    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9wt\" (UniqueName: \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.043458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.043329    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.043458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.043448    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.043593 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.043478    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.044212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.044183    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.046029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.046009    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.046115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.046079    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.105391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.105361    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9wt\" (UniqueName: \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\") pod \"maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.240116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.240030    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:25.570004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:25.569973    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:16:25.570630 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:16:25.570606    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55454ad_8f18_4afa_a3e5_b51944decbd2.slice/crio-6cfcef5324c3b8130f4c5d55e4f9166d534da669208c4e5ffa97c164a93cc9f3 WatchSource:0}: Error finding container 6cfcef5324c3b8130f4c5d55e4f9166d534da669208c4e5ffa97c164a93cc9f3: Status 404 returned error can't find the container with id 6cfcef5324c3b8130f4c5d55e4f9166d534da669208c4e5ffa97c164a93cc9f3
Oct 30 15:16:26.414048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:26.414009    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" event={"ID":"f55454ad-8f18-4afa-a3e5-b51944decbd2","Type":"ContainerStarted","Data":"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"}
Oct 30 15:16:26.414436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:26.414075    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" event={"ID":"f55454ad-8f18-4afa-a3e5-b51944decbd2","Type":"ContainerStarted","Data":"6cfcef5324c3b8130f4c5d55e4f9166d534da669208c4e5ffa97c164a93cc9f3"}
Oct 30 15:16:26.414436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:26.414104    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:26.470452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:26.465938    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" podStartSLOduration=1.952134707 podStartE2EDuration="2.465915283s" totalImagesPullingTime="513.780576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:26.463441506 +0000 UTC m=+1735.905332224" watchObservedRunningTime="2026-10-30 15:16:26.465915283 +0000 UTC m=+1735.907806000"
Oct 30 15:16:29.636509 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:29.636468    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:16:32.428426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:32.428391    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:16:37.068555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.068520    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:16:37.069109 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.068858    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerName="maas-api" containerID="cri-o://b1e074c405a483dc83b899f9dd33ccf6366a253609bd4d1b139b61b4ee168c1f" gracePeriod=30
Oct 30 15:16:37.355912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.355383    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:16:37.355912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.355655    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy" containerID="cri-o://6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237" gracePeriod=30
Oct 30 15:16:37.748561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.748308    5064 generic.go:356] "Generic (PLEG): container finished" podID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerID="b1e074c405a483dc83b899f9dd33ccf6366a253609bd4d1b139b61b4ee168c1f" exitCode=0
Oct 30 15:16:37.748561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.748386    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" event={"ID":"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e","Type":"ContainerDied","Data":"b1e074c405a483dc83b899f9dd33ccf6366a253609bd4d1b139b61b4ee168c1f"}
Oct 30 15:16:37.794686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.794659    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:16:37.896889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.896850    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\" (UniqueName: \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\") pod \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") "
Oct 30 15:16:37.897061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.896927    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\") pod \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") "
Oct 30 15:16:37.897061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.896968    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\") pod \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") "
Oct 30 15:16:37.897252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.897062    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\") pod \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\" (UID: \"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e\") "
Oct 30 15:16:37.897588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.897515    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca" pod "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" (UID: "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:16:37.898080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.898044    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:37.899348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.899319    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs" pod "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" (UID: "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:16:37.899348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.899337    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47" pod "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" (UID: "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e"). InnerVolumeSpecName "kube-api-access-vfh47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:37.899482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:37.899403    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls" pod "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" (UID: "e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:16:38.024913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.022991    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfh47\" (UniqueName: \"kubernetes.io/projected/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-kube-api-access-vfh47\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:38.046853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.023029    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:38.046853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.038919    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:38.758550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.758516    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"
Oct 30 15:16:38.759062 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.758549    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" event={"ID":"e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e","Type":"ContainerDied","Data":"677823a03b862255af4706b391693d959b18254be8386296fd95b23c98cc9cca"}
Oct 30 15:16:38.759062 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.758617    5064 scope.go:122] "RemoveContainer" containerID="b1e074c405a483dc83b899f9dd33ccf6366a253609bd4d1b139b61b4ee168c1f"
Oct 30 15:16:38.818152 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.818113    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:16:38.832136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:38.832105    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85"]
Oct 30 15:16:40.682470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:40.682430    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b7814-66c96984b5-5wt85" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.129:8443/health\": context deadline exceeded"
Oct 30 15:16:40.727702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:40.727016    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" path="/var/lib/kubelet/pods/e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e/volumes"
Oct 30 15:16:41.927138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.927099    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:16:41.928104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928079    5064 cpu_manager.go:414] "containerMap: removing container" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerName="maas-api"
Oct 30 15:16:41.928104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928103    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerName="maas-api"
Oct 30 15:16:41.928309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928170    5064 cpu_manager.go:414] "containerMap: removing container" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:41.928309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928180    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28085bb2-5d6a-43fe-b036-af6921999478" containerName="istio-proxy"
Oct 30 15:16:41.928309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928196    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:41.928309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928203    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:41.928509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928479    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00805fd-dea5-414c-8a93-4a7af0dc70df" containerName="istio-proxy"
Oct 30 15:16:41.928565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.928516    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f1b8ed-f8e8-4e4c-ad50-88a0249ecb0e" containerName="maas-api"
Oct 30 15:16:41.933694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.933673    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:41.958792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.958762    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:16:41.974135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:41.974103    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3bf5d2-serving-cert\""
Oct 30 15:16:42.027376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.027315    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwphx\" (UniqueName: \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.027900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.027870    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.028082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.028062    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.028185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.028126    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.131636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.131204    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwphx\" (UniqueName: \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.131636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.131414    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.131636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.131504    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.131636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.131584    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.132709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.132583    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.144768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.144378    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.181546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.181461    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.196711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.196663    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwphx\" (UniqueName: \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\") pod \"maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.252526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.252122    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:42.580758 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.580537    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:16:42.580898 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:16:42.580796    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93be9abd_9452_42f3_ab8c_78e0882fb9b9.slice/crio-d626b3882a9dd4d00018e1d666aee24d1490e0fc2f31221512f662934b40ea33 WatchSource:0}: Error finding container d626b3882a9dd4d00018e1d666aee24d1490e0fc2f31221512f662934b40ea33: Status 404 returned error can't find the container with id d626b3882a9dd4d00018e1d666aee24d1490e0fc2f31221512f662934b40ea33
Oct 30 15:16:42.639392 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.639364    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:16:42.755072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.754971    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755086    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755200    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755282    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755339    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755535 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755515    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755575    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755652    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755694    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\") pod \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\" (UID: \"59ccde36-91c4-4d10-99d1-0e7b185bfe21\") "
Oct 30 15:16:42.755790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755514    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:42.755913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755856    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:16:42.756232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.755976    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:42.756232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756196    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:42.756232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756208    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:42.756663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756644    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.756768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756668    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.756768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756682    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.756768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756695    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.756768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.756708    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.757900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.757880    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:16:42.758048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.758029    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:42.758444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.758428    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "kube-api-access-7k2xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:16:42.758501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.758474    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy" pod "59ccde36-91c4-4d10-99d1-0e7b185bfe21" (UID: "59ccde36-91c4-4d10-99d1-0e7b185bfe21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:16:42.831074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.831033    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" event={"ID":"93be9abd-9452-42f3-ab8c-78e0882fb9b9","Type":"ContainerStarted","Data":"d626b3882a9dd4d00018e1d666aee24d1490e0fc2f31221512f662934b40ea33"}
Oct 30 15:16:42.832953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.832926    5064 generic.go:356] "Generic (PLEG): container finished" podID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerID="6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237" exitCode=0
Oct 30 15:16:42.833072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.832986    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"
Oct 30 15:16:42.833072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.833011    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerDied","Data":"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"}
Oct 30 15:16:42.833072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.833063    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq" event={"ID":"59ccde36-91c4-4d10-99d1-0e7b185bfe21","Type":"ContainerDied","Data":"1de2f2468e1f860c0c74b8bb20b0eda73b56645d2e5bf2b817622da6d488e15d"}
Oct 30 15:16:42.833194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.833094    5064 scope.go:122] "RemoveContainer" containerID="6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"
Oct 30 15:16:42.845190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.845164    5064 scope.go:122] "RemoveContainer" containerID="8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"
Oct 30 15:16:42.870865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.867149    5064 scope.go:122] "RemoveContainer" containerID="6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"
Oct 30 15:16:42.889729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.888058    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.889729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.888096    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k2xj\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-kube-api-access-7k2xj\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.889729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.888111    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.889729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.888127    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ccde36-91c4-4d10-99d1-0e7b185bfe21-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:16:42.894031 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:42.893332    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237\": container with ID starting with 6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237 not found: ID does not exist" containerID="6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"
Oct 30 15:16:42.894031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.893475    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237"} err="failed to get container status \"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237\": rpc error: code = NotFound desc = could not find container \"6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237\": container with ID starting with 6ba2651b0e05dc231e259256cc46f89377a2dce47c15b8961a0a1eca74be8237 not found: ID does not exist"
Oct 30 15:16:42.894031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.893517    5064 scope.go:122] "RemoveContainer" containerID="8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"
Oct 30 15:16:42.894031 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:42.893922    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b\": container with ID starting with 8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b not found: ID does not exist" containerID="8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"
Oct 30 15:16:42.894031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.893950    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b"} err="failed to get container status \"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b\": rpc error: code = NotFound desc = could not find container \"8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b\": container with ID starting with 8e9aa456ff8e9b9d5d6d358f7c24c2e5547254026a43ae78865f5126a65c624b not found: ID does not exist"
Oct 30 15:16:42.940761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.939943    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:16:42.943116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:42.943088    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-gqdhq"]
Oct 30 15:16:43.843622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:43.843582    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" event={"ID":"93be9abd-9452-42f3-ab8c-78e0882fb9b9","Type":"ContainerStarted","Data":"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"}
Oct 30 15:16:43.843788 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:43.843679    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:43.892590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:43.892533    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" podStartSLOduration=2.512389174 podStartE2EDuration="2.89251783s" totalImagesPullingTime="380.128656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:43.889196386 +0000 UTC m=+1753.331087128" watchObservedRunningTime="2026-10-30 15:16:43.89251783 +0000 UTC m=+1753.334408547"
Oct 30 15:16:44.725747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:44.725715    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" path="/var/lib/kubelet/pods/59ccde36-91c4-4d10-99d1-0e7b185bfe21/volumes"
Oct 30 15:16:48.252134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.252099    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:16:48.252955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.252933    5064 cpu_manager.go:414] "containerMap: removing container" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:16:48.252955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.252954    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:16:48.253113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.252977    5064 cpu_manager.go:414] "containerMap: removing container" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:16:48.253113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.252983    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:16:48.253243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.253149    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:16:48.257049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.257030    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.276163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.276132    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:16:48.288995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.285610    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-765228-serving-cert\""
Oct 30 15:16:48.309677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.309617    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.309677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.309676    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkf4l\" (UniqueName: \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.310005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.309912    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.310005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.309993    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.411302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.411260    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.411302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.411303    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkf4l\" (UniqueName: \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.411512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.411463    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.411556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.411539    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.411599 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:48.411574    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-serving-cert: secret "maas-api-e2e-inf-b-765228-serving-cert" not found
Oct 30 15:16:48.411679 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:48.411663    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls podName:6ce7edba-0871-467f-9da5-54bdcf295ed0 nodeName:}" failed. No retries permitted until 2026-10-30 15:16:48.911639435 +0000 UTC m=+1758.353530145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls") pod "maas-api-e2e-inf-b-765228-74875597c8-ksnfr" (UID: "6ce7edba-0871-467f-9da5-54bdcf295ed0") : secret "maas-api-e2e-inf-b-765228-serving-cert" not found
Oct 30 15:16:48.412099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.412074    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.440887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.440753    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.442852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.442794    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkf4l\" (UniqueName: \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.948856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.948316    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:48.952415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:48.952385    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") pod \"maas-api-e2e-inf-b-765228-74875597c8-ksnfr\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:49.173574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:49.173534    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:49.557550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:49.557514    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:16:49.560052 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:16:49.560025    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce7edba_0871_467f_9da5_54bdcf295ed0.slice/crio-489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464 WatchSource:0}: Error finding container 489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464: Status 404 returned error can't find the container with id 489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464
Oct 30 15:16:49.874910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:49.874073    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:16:49.988626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:49.988589    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" event={"ID":"6ce7edba-0871-467f-9da5-54bdcf295ed0","Type":"ContainerStarted","Data":"489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464"}
Oct 30 15:16:50.776989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:50.775671    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:16:50.997820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:50.997782    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" event={"ID":"6ce7edba-0871-467f-9da5-54bdcf295ed0","Type":"ContainerStarted","Data":"fb856cc1dd0f3c3b544cc0cf612a2a02486ba22356b7bf3b95b343a2adf5b0c8"}
Oct 30 15:16:50.998243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:50.998222    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:51.061438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:51.061325    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" podStartSLOduration=2.643762136 podStartE2EDuration="3.061310594s" totalImagesPullingTime="417.548458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:51.059033955 +0000 UTC m=+1760.500924676" watchObservedRunningTime="2026-10-30 15:16:51.061310594 +0000 UTC m=+1760.503201311"
Oct 30 15:16:57.012930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.012890    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:16:57.055719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.055684    5064 generic.go:356] "Generic (PLEG): container finished" podID="c6991f3f-7592-4998-be53-204e55364fc3" containerID="462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3" exitCode=0
Oct 30 15:16:57.055921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.055764    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerDied","Data":"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"}
Oct 30 15:16:57.056285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.056267    5064 scope.go:122] "RemoveContainer" containerID="462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"
Oct 30 15:16:57.060783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.060730    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:16:57.060970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.060819    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:16:57.060970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:57.060886    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:16:58.065038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:58.064997    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerStarted","Data":"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"}
Oct 30 15:16:58.975010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:58.974977    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:16:58.975199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:58.975138    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:16:58.975269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:58.975193    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:16:59.651471 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:16:59.651424    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:16:59.975509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:59.975427    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:16:59.975509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:16:59.975491    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:17:00.976496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:00.976445    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:17:00.977051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:00.976539    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:17:01.975641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:01.975605    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:17:01.975889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:01.975674    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:17:02.974937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:02.974897    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:17:02.975342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:02.974987    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:17:03.975495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:03.975450    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 15:17:03.975993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:03.975527    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 15:17:04.991015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:04.990977    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:17:04.991531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:04.991414    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:17:04.992261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:04.992241    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:17:05.987074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:05.987033    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerName="main" containerID="cri-o://1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37" gracePeriod=45
Oct 30 15:17:06.020541 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.020501    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": dial tcp 10.132.0.133:8000: connect: connection refused"
Oct 30 15:17:06.278530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.278503    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:17:06.403618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.403584    5064 generic.go:356] "Generic (PLEG): container finished" podID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerID="1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37" exitCode=0
Oct 30 15:17:06.403806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.403628    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"
Oct 30 15:17:06.403806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.403674    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" event={"ID":"41354d05-6baf-43cd-8daf-60f03d07d59a","Type":"ContainerDied","Data":"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"}
Oct 30 15:17:06.403806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.403720    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6" event={"ID":"41354d05-6baf-43cd-8daf-60f03d07d59a","Type":"ContainerDied","Data":"ddcf803e10df63c9182dd6e92464dd19fe683703f29ed06ed5d1e7689dd1765d"}
Oct 30 15:17:06.403806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.403742    5064 scope.go:122] "RemoveContainer" containerID="1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"
Oct 30 15:17:06.412503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412473    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\" (UniqueName: \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412537    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412634    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\" (UniqueName: \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412706    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412775    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412805    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\") pod \"41354d05-6baf-43cd-8daf-60f03d07d59a\" (UID: \"41354d05-6baf-43cd-8daf-60f03d07d59a\") "
Oct 30 15:17:06.412974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.412845    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:17:06.413065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.413042    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:17:06.413128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.413098    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:17:06.413483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.413462    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.413483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.413483    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.413630 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.413498    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.415059 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.415037    5064 scope.go:122] "RemoveContainer" containerID="1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"
Oct 30 15:17:06.415299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.415257    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:17:06.415376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.415354    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:17:06.415439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.415395    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts" pod "41354d05-6baf-43cd-8daf-60f03d07d59a" (UID: "41354d05-6baf-43cd-8daf-60f03d07d59a"). InnerVolumeSpecName "kube-api-access-bttts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:17:06.415439 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:17:06.415408    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37\": container with ID starting with 1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37 not found: ID does not exist" containerID="1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"
Oct 30 15:17:06.415547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.415433    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37"} err="failed to get container status \"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37\": rpc error: code = NotFound desc = could not find container \"1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37\": container with ID starting with 1b248c119c4a6ba64bbbb33df056247c930137dcd326321bba59ba8c3a9bfe37 not found: ID does not exist"
Oct 30 15:17:06.515680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.515576    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41354d05-6baf-43cd-8daf-60f03d07d59a-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.515680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.515612    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41354d05-6baf-43cd-8daf-60f03d07d59a-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.515680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.515624    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttts\" (UniqueName: \"kubernetes.io/projected/41354d05-6baf-43cd-8daf-60f03d07d59a-kube-api-access-bttts\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:06.755776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.755722    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:17:06.769514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:06.769435    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24fff1/test-model-24fff1-kserve-77fd74bd97-qmgm6"]
Oct 30 15:17:08.725375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:08.725335    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" path="/var/lib/kubelet/pods/41354d05-6baf-43cd-8daf-60f03d07d59a/volumes"
Oct 30 15:17:29.656153 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:17:29.656088    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:17:31.096628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:31.096598    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:17:31.102587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:31.102567    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:17:38.154915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.154876    5064 generic.go:356] "Generic (PLEG): container finished" podID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerID="d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5" exitCode=0
Oct 30 15:17:38.154915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.154910    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerDied","Data":"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"}
Oct 30 15:17:38.155420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.155405    5064 scope.go:122] "RemoveContainer" containerID="d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"
Oct 30 15:17:38.157374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.157334    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:17:38.157473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.157406    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:17:38.157473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:38.157442    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:17:39.164674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:39.164636    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerStarted","Data":"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"}
Oct 30 15:17:39.845057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:39.845017    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:17:39.846790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:39.846757    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:39.846934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:39.846824    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:40.845652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:40.845612    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:40.846110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:40.845690    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:40.973478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:40.973441    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:17:41.845725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:41.845690    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:41.846122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:41.845747    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:42.929584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:42.929540    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:42.930224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:42.930187    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:43.845738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:43.845688    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:43.845983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:43.845757    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:44.845631 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:44.845589    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:17:44.846083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:44.845658    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:17:45.852639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:45.852602    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:17:45.854050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:45.854028    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:17:56.250375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.250335    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:17:56.250855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.250604    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" podUID="c51166b0-5af4-445b-a420-470b3bf24090" containerName="maas-api" containerID="cri-o://18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5" gracePeriod=30
Oct 30 15:17:56.551501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.551474    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:17:56.631862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.621317    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\") pod \"c51166b0-5af4-445b-a420-470b3bf24090\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") "
Oct 30 15:17:56.636297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.636265    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca" pod "c51166b0-5af4-445b-a420-470b3bf24090" (UID: "c51166b0-5af4-445b-a420-470b3bf24090"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:17:56.636964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.636897    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\" (UniqueName: \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\") pod \"c51166b0-5af4-445b-a420-470b3bf24090\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") "
Oct 30 15:17:56.637377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.637357    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\") pod \"c51166b0-5af4-445b-a420-470b3bf24090\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") "
Oct 30 15:17:56.638332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.638312    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\") pod \"c51166b0-5af4-445b-a420-470b3bf24090\" (UID: \"c51166b0-5af4-445b-a420-470b3bf24090\") "
Oct 30 15:17:56.680135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.680066    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls" pod "c51166b0-5af4-445b-a420-470b3bf24090" (UID: "c51166b0-5af4-445b-a420-470b3bf24090"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:17:56.680853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.680809    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:56.681017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.681003    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51166b0-5af4-445b-a420-470b3bf24090-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:56.688195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.688141    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8" pod "c51166b0-5af4-445b-a420-470b3bf24090" (UID: "c51166b0-5af4-445b-a420-470b3bf24090"). InnerVolumeSpecName "kube-api-access-gvtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:17:56.693464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.693413    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs" pod "c51166b0-5af4-445b-a420-470b3bf24090" (UID: "c51166b0-5af4-445b-a420-470b3bf24090"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:17:56.792936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.791432    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtm8\" (UniqueName: \"kubernetes.io/projected/c51166b0-5af4-445b-a420-470b3bf24090-kube-api-access-gvtm8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:56.792936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.791478    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c51166b0-5af4-445b-a420-470b3bf24090-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:56.985301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.985265    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:17:56.985741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:56.985712    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" podUID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" containerName="maas-api" containerID="cri-o://1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274" gracePeriod=30
Oct 30 15:17:57.307782 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.307755    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:17:57.339537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.339445    5064 generic.go:356] "Generic (PLEG): container finished" podID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" containerID="1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274" exitCode=0
Oct 30 15:17:57.339537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.339503    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"
Oct 30 15:17:57.339761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.339528    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" event={"ID":"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab","Type":"ContainerDied","Data":"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"}
Oct 30 15:17:57.339761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.339584    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d" event={"ID":"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab","Type":"ContainerDied","Data":"7db1e34ebf5f57d0405c2fd8db0af0fb0138535918aeb03cc1c9e3339e599f13"}
Oct 30 15:17:57.339761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.339612    5064 scope.go:122] "RemoveContainer" containerID="1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"
Oct 30 15:17:57.341263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.341242    5064 generic.go:356] "Generic (PLEG): container finished" podID="c51166b0-5af4-445b-a420-470b3bf24090" containerID="18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5" exitCode=0
Oct 30 15:17:57.341394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.341301    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"
Oct 30 15:17:57.341394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.341304    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" event={"ID":"c51166b0-5af4-445b-a420-470b3bf24090","Type":"ContainerDied","Data":"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"}
Oct 30 15:17:57.341394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.341364    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm" event={"ID":"c51166b0-5af4-445b-a420-470b3bf24090","Type":"ContainerDied","Data":"4690b09ee6c60d2aa41e03842227efd56929a3343b1ec5c13dcab9292940e405"}
Oct 30 15:17:57.352809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.352756    5064 scope.go:122] "RemoveContainer" containerID="1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"
Oct 30 15:17:57.353233 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:17:57.353191    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274\": container with ID starting with 1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274 not found: ID does not exist" containerID="1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"
Oct 30 15:17:57.353233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.353232    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274"} err="failed to get container status \"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274\": rpc error: code = NotFound desc = could not find container \"1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274\": container with ID starting with 1a50a7ad20d97a74137c2de887883c248f352ce81e4f3fd411a1de397b976274 not found: ID does not exist"
Oct 30 15:17:57.353233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.353259    5064 scope.go:122] "RemoveContainer" containerID="18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"
Oct 30 15:17:57.363575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.363546    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\" (UniqueName: \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\") pod \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") "
Oct 30 15:17:57.363702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.363652    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\") pod \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") "
Oct 30 15:17:57.363813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.363797    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") pod \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") "
Oct 30 15:17:57.364225 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.363974    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\") pod \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\" (UID: \"ff523d8c-f1e6-4e98-89eb-4f1ccb706dab\") "
Oct 30 15:17:57.364382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.364355    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca" pod "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" (UID: "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:17:57.364804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.364782    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:57.366537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.366508    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs" pod "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" (UID: "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:17:57.366643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.366564    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls" pod "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" (UID: "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:17:57.369363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.369340    5064 scope.go:122] "RemoveContainer" containerID="18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"
Oct 30 15:17:57.369703 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:17:57.369673    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5\": container with ID starting with 18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5 not found: ID does not exist" containerID="18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"
Oct 30 15:17:57.369783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.369714    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5"} err="failed to get container status \"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5\": rpc error: code = NotFound desc = could not find container \"18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5\": container with ID starting with 18ffbf6de765b7a4668a636ff95a4cb39ae85233ec068fbbf04b2b3a4e8480e5 not found: ID does not exist"
Oct 30 15:17:57.370220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.370197    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2" pod "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" (UID: "ff523d8c-f1e6-4e98-89eb-4f1ccb706dab"). InnerVolumeSpecName "kube-api-access-2dvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:17:57.417083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.417030    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:17:57.418240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.418204    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-751ab3-84884646b9-2dndm"]
Oct 30 15:17:57.466131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.466095    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvn2\" (UniqueName: \"kubernetes.io/projected/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-kube-api-access-2dvn2\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:57.466131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.466126    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:57.466131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.466136    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:17:57.800855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.796895    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:17:57.804324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:57.803244    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c69f40-576ff889bf-4bx2d"]
Oct 30 15:17:58.725186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:58.725148    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51166b0-5af4-445b-a420-470b3bf24090" path="/var/lib/kubelet/pods/c51166b0-5af4-445b-a420-470b3bf24090/volumes"
Oct 30 15:17:58.725695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:58.725679    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" path="/var/lib/kubelet/pods/ff523d8c-f1e6-4e98-89eb-4f1ccb706dab/volumes"
Oct 30 15:17:59.347190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.347153    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:17:59.347432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.347408    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" podUID="91cbf1a8-167f-48bc-9b5b-c4113791a223" containerName="maas-api" containerID="cri-o://e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04" gracePeriod=30
Oct 30 15:17:59.917083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.917050    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:17:59.988539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.988441    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\") pod \"91cbf1a8-167f-48bc-9b5b-c4113791a223\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") "
Oct 30 15:17:59.989055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.989026    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\") pod \"91cbf1a8-167f-48bc-9b5b-c4113791a223\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") "
Oct 30 15:18:00.011915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:17:59.997991    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\" (UniqueName: \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\") pod \"91cbf1a8-167f-48bc-9b5b-c4113791a223\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") "
Oct 30 15:18:00.012380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.012352    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\") pod \"91cbf1a8-167f-48bc-9b5b-c4113791a223\" (UID: \"91cbf1a8-167f-48bc-9b5b-c4113791a223\") "
Oct 30 15:18:00.049340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.016284    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca" pod "91cbf1a8-167f-48bc-9b5b-c4113791a223" (UID: "91cbf1a8-167f-48bc-9b5b-c4113791a223"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:18:00.049340 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:00.021179    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:18:00.049340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.021487    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls" pod "91cbf1a8-167f-48bc-9b5b-c4113791a223" (UID: "91cbf1a8-167f-48bc-9b5b-c4113791a223"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:18:00.049340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.040469    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs" pod "91cbf1a8-167f-48bc-9b5b-c4113791a223" (UID: "91cbf1a8-167f-48bc-9b5b-c4113791a223"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:18:00.050795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.050765    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:00.050987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.050972    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cbf1a8-167f-48bc-9b5b-c4113791a223-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:00.051096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.051084    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cbf1a8-167f-48bc-9b5b-c4113791a223-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:00.062941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.052879    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8" pod "91cbf1a8-167f-48bc-9b5b-c4113791a223" (UID: "91cbf1a8-167f-48bc-9b5b-c4113791a223"). InnerVolumeSpecName "kube-api-access-898n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:18:00.152736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.152698    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-898n8\" (UniqueName: \"kubernetes.io/projected/91cbf1a8-167f-48bc-9b5b-c4113791a223-kube-api-access-898n8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:00.378169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.378137    5064 generic.go:356] "Generic (PLEG): container finished" podID="91cbf1a8-167f-48bc-9b5b-c4113791a223" containerID="e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04" exitCode=0
Oct 30 15:18:00.378309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.378206    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"
Oct 30 15:18:00.378309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.378262    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" event={"ID":"91cbf1a8-167f-48bc-9b5b-c4113791a223","Type":"ContainerDied","Data":"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"}
Oct 30 15:18:00.378398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.378317    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv" event={"ID":"91cbf1a8-167f-48bc-9b5b-c4113791a223","Type":"ContainerDied","Data":"14c28174cd171085ae68fe7e936f6d7345299812f6cb8605a38fc524e2998ac2"}
Oct 30 15:18:00.378398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.378346    5064 scope.go:122] "RemoveContainer" containerID="e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"
Oct 30 15:18:00.390331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.390314    5064 scope.go:122] "RemoveContainer" containerID="e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"
Oct 30 15:18:00.390638 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:00.390620    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04\": container with ID starting with e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04 not found: ID does not exist" containerID="e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"
Oct 30 15:18:00.390683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.390649    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04"} err="failed to get container status \"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04\": rpc error: code = NotFound desc = could not find container \"e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04\": container with ID starting with e515049ae74bb9543fe7087b77b36b75863eaeb96e565685738f726a68093f04 not found: ID does not exist"
Oct 30 15:18:00.443974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.443820    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:18:00.450989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.450938    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fff1-f47959d89-dppmv"]
Oct 30 15:18:00.726608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:00.726514    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cbf1a8-167f-48bc-9b5b-c4113791a223" path="/var/lib/kubelet/pods/91cbf1a8-167f-48bc-9b5b-c4113791a223/volumes"
Oct 30 15:18:04.157683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.157641    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:18:04.158897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158875    5064 cpu_manager.go:414] "containerMap: removing container" podUID="91cbf1a8-167f-48bc-9b5b-c4113791a223" containerName="maas-api"
Oct 30 15:18:04.158973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158900    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cbf1a8-167f-48bc-9b5b-c4113791a223" containerName="maas-api"
Oct 30 15:18:04.158973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158956    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c51166b0-5af4-445b-a420-470b3bf24090" containerName="maas-api"
Oct 30 15:18:04.158973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158967    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51166b0-5af4-445b-a420-470b3bf24090" containerName="maas-api"
Oct 30 15:18:04.159072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158985    5064 cpu_manager.go:414] "containerMap: removing container" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerName="main"
Oct 30 15:18:04.159072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.158994    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerName="main"
Oct 30 15:18:04.159072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159014    5064 cpu_manager.go:414] "containerMap: removing container" podUID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" containerName="maas-api"
Oct 30 15:18:04.159072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159021    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" containerName="maas-api"
Oct 30 15:18:04.159203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159191    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ccde36-91c4-4d10-99d1-0e7b185bfe21" containerName="istio-proxy"
Oct 30 15:18:04.159237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159215    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cbf1a8-167f-48bc-9b5b-c4113791a223" containerName="maas-api"
Oct 30 15:18:04.159237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159231    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51166b0-5af4-445b-a420-470b3bf24090" containerName="maas-api"
Oct 30 15:18:04.159300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159244    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="41354d05-6baf-43cd-8daf-60f03d07d59a" containerName="main"
Oct 30 15:18:04.159300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.159258    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff523d8c-f1e6-4e98-89eb-4f1ccb706dab" containerName="maas-api"
Oct 30 15:18:04.165646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.165622    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.189892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.188778    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:18:04.191162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.191133    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d84bc6-serving-cert\""
Oct 30 15:18:04.227412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.227381    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.227587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.227439    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.227587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.227561    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.227717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.227699    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjbg\" (UniqueName: \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.363874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.363211    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.363874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.363297    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.363874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.363344    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.363874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.363556    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjbg\" (UniqueName: \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.374984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.374389    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.374984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.374797    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.402854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.401617    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.432645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.427947    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjbg\" (UniqueName: \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\") pod \"maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.501222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.501184    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:04.815211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:04.815174    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:18:04.815665 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:18:04.815638    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5130c9cf_658b_44d1_92f9_53f8a21924a4.slice/crio-f910f16603f1294658437796dd6454069b42826f96373f8ef3ae24a32e005888 WatchSource:0}: Error finding container f910f16603f1294658437796dd6454069b42826f96373f8ef3ae24a32e005888: Status 404 returned error can't find the container with id f910f16603f1294658437796dd6454069b42826f96373f8ef3ae24a32e005888
Oct 30 15:18:05.450932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:05.450810    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" event={"ID":"5130c9cf-658b-44d1-92f9-53f8a21924a4","Type":"ContainerStarted","Data":"6d246255201e2ea19e025d8897fdff65cf6d699b73a49e038401a2fa6fd7dfe1"}
Oct 30 15:18:05.450932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:05.450881    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" event={"ID":"5130c9cf-658b-44d1-92f9-53f8a21924a4","Type":"ContainerStarted","Data":"f910f16603f1294658437796dd6454069b42826f96373f8ef3ae24a32e005888"}
Oct 30 15:18:05.450932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:05.450903    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:05.505449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:05.505384    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" podStartSLOduration=1.122731563 podStartE2EDuration="1.505366927s" totalImagesPullingTime="382.635364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:05.501796493 +0000 UTC m=+1834.943687210" watchObservedRunningTime="2026-10-30 15:18:05.505366927 +0000 UTC m=+1834.947257645"
Oct 30 15:18:07.772385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.772342    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:18:07.778661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.778638    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.799571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.799542    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:18:07.809896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.809863    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810046    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810120    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810148    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810175    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810335    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810397    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkd8\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810497    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.810556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.810543    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.828138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.828108    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a9275a-openshift-default-dockercfg-pkhdl\""
Oct 30 15:18:07.929167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929001    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929121    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929184    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929334    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929399    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkd8\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929484    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929531    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.929752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.929582    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.964855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.930229    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.964855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.932528    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.964855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.932815    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:07.966342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:07.966304    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.047932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.002997    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.063934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.063888    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.086853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.064478    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkd8\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.086853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.072627    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.158454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.158419    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.397444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.397415    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:08.810219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.810169    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:18:08.810669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.810270    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:18:08.810669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.810316    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:18:08.875875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:08.871735    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:18:09.935733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:09.934415    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:18:09.935733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:09.934653    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" containerID="cri-o://a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f" gracePeriod=30
Oct 30 15:18:09.951578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:09.951525    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerStarted","Data":"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"}
Oct 30 15:18:09.951780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:09.951610    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerStarted","Data":"d517ab93e37e7b175d9cd2c452e1552904a3d9e20371d8391f1566753d0092a1"}
Oct 30 15:18:10.055504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:10.055446    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podStartSLOduration=3.055431023 podStartE2EDuration="3.055431023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:10.05219861 +0000 UTC m=+1839.494089327" watchObservedRunningTime="2026-10-30 15:18:10.055431023 +0000 UTC m=+1839.497321739"
Oct 30 15:18:10.397752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:10.397722    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:10.404077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:10.404049    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:10.959592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:10.959562    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:10.960486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:10.960464    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:18:11.311510 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.309392    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:18:11.319856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.319591    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.326712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.326675    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:18:11.341012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.340985    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765228\"/\"openshift-service-ca.crt\""
Oct 30 15:18:11.341012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.340999    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765228\"/\"kube-root-ca.crt\""
Oct 30 15:18:11.365276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.365247    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765228\"/\"default-dockercfg-pshd8\""
Oct 30 15:18:11.365455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.365316    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765228\"/\"test-model-765228-kserve-self-signed-certs\""
Oct 30 15:18:11.428211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428176    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.428211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428214    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.428428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428323    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.428490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428463    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsbl\" (UniqueName: \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.428638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428614    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.428693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.428680    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.467288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.467259    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:18:11.488878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.488813    5064 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:18:11.489050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.488889    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544214    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544476    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsbl\" (UniqueName: \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544660    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544736    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544945    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.548879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.544976    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.571902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.571218    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.581913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.572221    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.618855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.598628    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.728066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.728035    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsbl\" (UniqueName: \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.728405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.728376    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.733311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.733270    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\") pod \"test-model-765228-kserve-7ccf64cb99-xbcrx\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") " pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:11.946072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:11.946038    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:12.264550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:12.264523    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:18:12.265217 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:18:12.265184    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5a168d_a066_4694_8225_a3f7fe95eb8f.slice/crio-b21e45df693cc6c05518bc8281b2cd0f14ed32da7945bc4d39bbe007fc11b026 WatchSource:0}: Error finding container b21e45df693cc6c05518bc8281b2cd0f14ed32da7945bc4d39bbe007fc11b026: Status 404 returned error can't find the container with id b21e45df693cc6c05518bc8281b2cd0f14ed32da7945bc4d39bbe007fc11b026
Oct 30 15:18:12.981656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:12.981609    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" event={"ID":"1a5a168d-a066-4694-8225-a3f7fe95eb8f","Type":"ContainerStarted","Data":"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"}
Oct 30 15:18:12.981882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:12.981662    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" event={"ID":"1a5a168d-a066-4694-8225-a3f7fe95eb8f","Type":"ContainerStarted","Data":"b21e45df693cc6c05518bc8281b2cd0f14ed32da7945bc4d39bbe007fc11b026"}
Oct 30 15:18:13.042439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:13.042379    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" podStartSLOduration=2.04236316 podStartE2EDuration="2.04236316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:13.038400319 +0000 UTC m=+1842.480291036" watchObservedRunningTime="2026-10-30 15:18:13.04236316 +0000 UTC m=+1842.484253877"
Oct 30 15:18:13.946944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:13.946907    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:15.460329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.460296    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573282    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573450    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573559    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573634    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573664    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573692    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573754    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573949    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.573993    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\") pod \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\" (UID: \"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43\") "
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.576225    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:15.576968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.576652    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:18:15.578004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.577751    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:15.578004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.577973    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:15.637045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.634458    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:15.640036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.638842    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:18:15.640260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.640224    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:18:15.640674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.640643    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:15.641458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.641429    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq" pod "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" (UID: "4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43"). InnerVolumeSpecName "kube-api-access-2lmmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:18:15.676528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676496    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676523    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676533    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676542    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmmq\" (UniqueName: \"kubernetes.io/projected/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-kube-api-access-2lmmq\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676551    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676569    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676578    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676587    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:15.676888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:15.676594    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:16.278933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.278894    5064 generic.go:356] "Generic (PLEG): container finished" podID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerID="a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f" exitCode=0
Oct 30 15:18:16.279124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.278967    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"
Oct 30 15:18:16.279124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.278988    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerDied","Data":"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"}
Oct 30 15:18:16.279124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.279071    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg" event={"ID":"4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43","Type":"ContainerDied","Data":"ef1aaf62c3db20c486f2e8bf451568203b9a126f393f10bfc6e0725ddaf11e82"}
Oct 30 15:18:16.279124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.279110    5064 scope.go:122] "RemoveContainer" containerID="a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"
Oct 30 15:18:16.292851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.292812    5064 scope.go:122] "RemoveContainer" containerID="c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"
Oct 30 15:18:16.305099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.305077    5064 scope.go:122] "RemoveContainer" containerID="a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"
Oct 30 15:18:16.305541 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:16.305515    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f\": container with ID starting with a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f not found: ID does not exist" containerID="a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"
Oct 30 15:18:16.305627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.305554    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f"} err="failed to get container status \"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f\": rpc error: code = NotFound desc = could not find container \"a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f\": container with ID starting with a533805e62323a207db2bd58c56e6f4d2d102605f0ecc99a810c30751b9c6a8f not found: ID does not exist"
Oct 30 15:18:16.305627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.305582    5064 scope.go:122] "RemoveContainer" containerID="c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"
Oct 30 15:18:16.305895 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:16.305866    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1\": container with ID starting with c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1 not found: ID does not exist" containerID="c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"
Oct 30 15:18:16.305978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.305905    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1"} err="failed to get container status \"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1\": rpc error: code = NotFound desc = could not find container \"c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1\": container with ID starting with c7ffd12766fd400b727df48dafe4ff7b6e4523b8c0b73512b9611f5590fdfae1 not found: ID does not exist"
Oct 30 15:18:16.354637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.354601    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:18:16.368498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.368458    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-thhwg"]
Oct 30 15:18:16.725704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:16.725666    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" path="/var/lib/kubelet/pods/4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43/volumes"
Oct 30 15:18:21.946237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:21.946201    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:21.959428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:21.959403    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:22.341954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:22.341922    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:18:29.632304 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:29.632262    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:18:31.309665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:31.309629    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:18:31.310140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:31.309933    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy" containerID="cri-o://63e55d0724fc368296b5966be3386322746a51d262d5af517eb70be7ca5c6152" gracePeriod=30
Oct 30 15:18:31.785151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:31.785114    5064 scope.go:122] "RemoveContainer" containerID="e38b1a15bbcb5b0f93d1ac63e5efd4a41ca3b19ba8c1783655267c7159f94bde"
Oct 30 15:18:36.536137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.536095    5064 generic.go:356] "Generic (PLEG): container finished" podID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerID="63e55d0724fc368296b5966be3386322746a51d262d5af517eb70be7ca5c6152" exitCode=0
Oct 30 15:18:36.536610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.536134    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerDied","Data":"63e55d0724fc368296b5966be3386322746a51d262d5af517eb70be7ca5c6152"}
Oct 30 15:18:36.536610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.536218    5064 scope.go:122] "RemoveContainer" containerID="0df976cd18230e7e3367b2fb03c8d5ece89c9136ce2194d30eac6d8b8bc9e83e"
Oct 30 15:18:36.684861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.684350    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:18:36.800350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800293    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800349    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800494    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800581    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800670    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800672    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:36.800868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800728    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.800868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800795    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.801072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800939    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.801072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800969    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:36.801072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.800990    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:18:36.801309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.801098    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\") pod \"fdd26fed-403a-410f-9bd6-daec9ae621b9\" (UID: \"fdd26fed-403a-410f-9bd6-daec9ae621b9\") "
Oct 30 15:18:36.801309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.801230    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:36.802755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.802111    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.802755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.802139    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd26fed-403a-410f-9bd6-daec9ae621b9-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.802755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.802154    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.802755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.802168    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.802755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.802386    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:36.804530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.804502    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "kube-api-access-jn9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:18:36.804643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.804618    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:18:36.806453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.806425    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:18:36.807584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.807563    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token" pod "fdd26fed-403a-410f-9bd6-daec9ae621b9" (UID: "fdd26fed-403a-410f-9bd6-daec9ae621b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:18:36.903468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.903432    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.903468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.903465    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9t5\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-kube-api-access-jn9t5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.903468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.903476    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd26fed-403a-410f-9bd6-daec9ae621b9-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.903755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.903485    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:36.903755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:36.903495    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd26fed-403a-410f-9bd6-daec9ae621b9-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:18:37.630753 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:37.630714    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82" event={"ID":"fdd26fed-403a-410f-9bd6-daec9ae621b9","Type":"ContainerDied","Data":"0b51da3831c51c7f90ff2a47027857d8f88df5b52e9be8c3ac41aed64d4c5712"}
Oct 30 15:18:37.631461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:37.631431    5064 scope.go:122] "RemoveContainer" containerID="63e55d0724fc368296b5966be3386322746a51d262d5af517eb70be7ca5c6152"
Oct 30 15:18:37.631712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:37.631333    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"
Oct 30 15:18:37.836852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:37.836174    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:18:37.860851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:37.859155    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-47bcfe-openshift-default-c5b44f8d5-t7w82"]
Oct 30 15:18:38.739377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:38.739335    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" path="/var/lib/kubelet/pods/fdd26fed-403a-410f-9bd6-daec9ae621b9/volumes"
Oct 30 15:18:49.624666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.624620    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:18:49.625876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625825    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:49.625876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625862    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:49.625876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625877    5064 cpu_manager.go:414] "containerMap: removing container" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:49.626036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625886    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:49.626036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625919    5064 cpu_manager.go:414] "containerMap: removing container" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:49.626036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625927    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:49.626036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625972    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:49.626036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.625982    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:49.626192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.626153    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:49.626192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.626165    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:49.632377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.632352    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.649482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.649449    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:18:49.658307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.658266    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"kube-root-ca.crt\""
Oct 30 15:18:49.658307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.658266    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"openshift-service-ca.crt\""
Oct 30 15:18:49.664024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.663999    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"default-dockercfg-4frl5\""
Oct 30 15:18:49.664174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.664123    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"auth-test-model-100b2e-kserve-self-signed-certs\""
Oct 30 15:18:49.740768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.740731    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.741021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.740884    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.741021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.740978    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.741021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.741016    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.741199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.741082    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l789\" (UniqueName: \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.741199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.741114    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.842475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.842441    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.842740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.842703    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.842904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.842849    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843083    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843190    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843424    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843459    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.843738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.843590    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l789\" (UniqueName: \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.845523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.845500    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.845771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.845753    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.895896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.891882    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l789\" (UniqueName: \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\") pod \"auth-test-model-100b2e-kserve-5bccd76f65-fjbht\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:49.957856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:49.954102    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:50.372765 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:18:50.372724    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod365e40cd_e16f_4df8_a5fa_dcb7dcafd352.slice/crio-4d2c6a24a943ec9c4527e8ceba77c5870e616491b9c4483c032db8425ddbc8ab WatchSource:0}: Error finding container 4d2c6a24a943ec9c4527e8ceba77c5870e616491b9c4483c032db8425ddbc8ab: Status 404 returned error can't find the container with id 4d2c6a24a943ec9c4527e8ceba77c5870e616491b9c4483c032db8425ddbc8ab
Oct 30 15:18:50.373336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:50.373305    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:18:50.750028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:50.749934    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" event={"ID":"365e40cd-e16f-4df8-a5fa-dcb7dcafd352","Type":"ContainerStarted","Data":"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"}
Oct 30 15:18:50.750028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:50.749983    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" event={"ID":"365e40cd-e16f-4df8-a5fa-dcb7dcafd352","Type":"ContainerStarted","Data":"4d2c6a24a943ec9c4527e8ceba77c5870e616491b9c4483c032db8425ddbc8ab"}
Oct 30 15:18:50.795873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:50.795766    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" podStartSLOduration=1.795743534 podStartE2EDuration="1.795743534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:50.794689623 +0000 UTC m=+1880.236580340" watchObservedRunningTime="2026-10-30 15:18:50.795743534 +0000 UTC m=+1880.237634251"
Oct 30 15:18:50.955276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:50.955244    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:55.067633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.067599    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:18:55.068502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.068484    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6b9b60-8ec4-41c9-9bb6-1fcedf711d43" containerName="istio-proxy"
Oct 30 15:18:55.068561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.068506    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd26fed-403a-410f-9bd6-daec9ae621b9" containerName="istio-proxy"
Oct 30 15:18:55.072355 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.072336    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.092380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.092343    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:18:55.100436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.100408    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a9275a-serving-cert\""
Oct 30 15:18:55.173682 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.173628    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.173918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.173898    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.174067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.174048    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.174294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.174276    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbvt\" (UniqueName: \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.275787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.275726    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.276024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.275956    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.276207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.276185    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbvt\" (UniqueName: \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.276280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.276228    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.276887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.276825    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.278573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.278540    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.278697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.278609    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.327903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.318702    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbvt\" (UniqueName: \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\") pod \"maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.391543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.390820    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:55.714880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.709297    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:18:55.714880 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:18:55.711073    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb241ae0a_71c3_4294_8580_a2d0d42b4b56.slice/crio-72b3d778bfeb1b076e955520a87cca77016c1ee70984930bd3943be9e462dd45 WatchSource:0}: Error finding container 72b3d778bfeb1b076e955520a87cca77016c1ee70984930bd3943be9e462dd45: Status 404 returned error can't find the container with id 72b3d778bfeb1b076e955520a87cca77016c1ee70984930bd3943be9e462dd45
Oct 30 15:18:55.793751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:55.793709    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" event={"ID":"b241ae0a-71c3-4294-8580-a2d0d42b4b56","Type":"ContainerStarted","Data":"72b3d778bfeb1b076e955520a87cca77016c1ee70984930bd3943be9e462dd45"}
Oct 30 15:18:56.802846 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:56.802794    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" event={"ID":"b241ae0a-71c3-4294-8580-a2d0d42b4b56","Type":"ContainerStarted","Data":"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"}
Oct 30 15:18:56.803230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:56.802874    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:18:56.848720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:56.848667    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" podStartSLOduration=1.510710718 podStartE2EDuration="1.848651932s" totalImagesPullingTime="337.941214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:56.847620663 +0000 UTC m=+1886.289511381" watchObservedRunningTime="2026-10-30 15:18:56.848651932 +0000 UTC m=+1886.290542648"
Oct 30 15:18:59.632905 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:18:59.632868    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:18:59.954712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:59.954625    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:18:59.967964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:18:59.967938    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:19:00.857765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:00.857734    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:19:02.816403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:02.816371    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:19:18.111001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.110951    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:19:18.118479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.118451    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.134465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.134433    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:19:18.144048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.142522    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34218d\"/\"default-dockercfg-fk8gn\""
Oct 30 15:19:18.144048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.142612    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34218d\"/\"test-model-34218d-kserve-self-signed-certs\""
Oct 30 15:19:18.149449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.149421    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34218d\"/\"openshift-service-ca.crt\""
Oct 30 15:19:18.149449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.149441    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34218d\"/\"kube-root-ca.crt\""
Oct 30 15:19:18.158655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.158630    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.158797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.158667    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtgf\" (UniqueName: \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.158909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.158874    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.159075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.159055    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.159178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.159162    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.159286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.159272    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.260919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.260874    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.262248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.262211    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.262461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.262359    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.262461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.262404    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.262461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.262441    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtgf\" (UniqueName: \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.262655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.262547    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.263863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.263816    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.265337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.264335    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.265515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.264595    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.278183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.278137    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.279282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.279250    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.290358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.290325    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtgf\" (UniqueName: \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\") pod \"test-model-34218d-kserve-67cd556df9-smhkv\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") " pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.440179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.440137    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:18.878506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:18.878464    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:19:18.883243 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:19:18.879912    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a93b058_4eed_4e3d_b397_192ea837fec6.slice/crio-2248534ca10c203591b9129935ea29da6097360f8c9375e19f7b0a93c596b5e7 WatchSource:0}: Error finding container 2248534ca10c203591b9129935ea29da6097360f8c9375e19f7b0a93c596b5e7: Status 404 returned error can't find the container with id 2248534ca10c203591b9129935ea29da6097360f8c9375e19f7b0a93c596b5e7
Oct 30 15:19:19.089594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:19.089547    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" event={"ID":"0a93b058-4eed-4e3d-b397-192ea837fec6","Type":"ContainerStarted","Data":"ff1d4cf1a015b604f25a496eac4f576d81a2d96191ff4d95456c61dc4660f15a"}
Oct 30 15:19:19.089808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:19.089647    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" event={"ID":"0a93b058-4eed-4e3d-b397-192ea837fec6","Type":"ContainerStarted","Data":"2248534ca10c203591b9129935ea29da6097360f8c9375e19f7b0a93c596b5e7"}
Oct 30 15:19:19.164737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:19.163235    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" podStartSLOduration=1.163212254 podStartE2EDuration="1.163212254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:19.158107606 +0000 UTC m=+1908.599998322" watchObservedRunningTime="2026-10-30 15:19:19.163212254 +0000 UTC m=+1908.605102972"
Oct 30 15:19:19.444102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:19.444005    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:21.890782 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.890730    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:19:21.895963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.895936    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.956790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.956750    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:19:21.989333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989295    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989380    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989483    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989505    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989525    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989545    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989571    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.989739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989613    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nbnh\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.990046 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.989867    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:21.997750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:21.997725    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a4f54-openshift-default-dockercfg-8tmv4\""
Oct 30 15:19:22.091494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091397    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091631    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091654    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091677    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091701    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091720    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091747    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.091943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.091770    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nbnh\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.092526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.092492    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.092795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.092772    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.093274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.093202    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.093274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.093264    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.093514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.093493    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.095459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.095431    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.095632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.095613    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.142094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.142043    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.143397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.143367    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nbnh\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.248189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.248154    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:22.594228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.594030    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:19:22.594379 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:19:22.594269    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f2b11c_2539_436a_8264_c0db6dfb87a0.slice/crio-8e4ec2c9b4c330871c704d4fbdbd4671906ec9b30a041bf13cc5ccaa59879a39 WatchSource:0}: Error finding container 8e4ec2c9b4c330871c704d4fbdbd4671906ec9b30a041bf13cc5ccaa59879a39: Status 404 returned error can't find the container with id 8e4ec2c9b4c330871c704d4fbdbd4671906ec9b30a041bf13cc5ccaa59879a39
Oct 30 15:19:22.597458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.597421    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:22.597572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.597511    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:22.597572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:22.597557    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:23.154337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.154299    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" event={"ID":"d4f2b11c-2539-436a-8264-c0db6dfb87a0","Type":"ContainerStarted","Data":"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"}
Oct 30 15:19:23.155327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.155297    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" event={"ID":"d4f2b11c-2539-436a-8264-c0db6dfb87a0","Type":"ContainerStarted","Data":"8e4ec2c9b4c330871c704d4fbdbd4671906ec9b30a041bf13cc5ccaa59879a39"}
Oct 30 15:19:23.246719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.246650    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podStartSLOduration=2.246627191 podStartE2EDuration="2.246627191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:23.220355267 +0000 UTC m=+1912.662245985" watchObservedRunningTime="2026-10-30 15:19:23.246627191 +0000 UTC m=+1912.688517910"
Oct 30 15:19:23.249256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.249215    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:23.251280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.251243    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:23.251417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:23.251368    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:24.263857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:24.250395    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:24.263857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:24.250474    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:25.249102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.249065    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:25.249351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.249125    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:25.698149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.698116    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:19:25.703741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.703704    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.707389    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tjd\" (UniqueName: \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.707478    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.707581    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.707824    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.708897    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.711763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.708979    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.747910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.738737    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3bf5d2\"/\"openshift-service-ca.crt\""
Oct 30 15:19:25.747910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.738913    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3bf5d2\"/\"kube-root-ca.crt\""
Oct 30 15:19:25.747910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.743432    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3bf5d2\"/\"test-model-3bf5d2-kserve-self-signed-certs\""
Oct 30 15:19:25.777887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.777848    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:19:25.811078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.811036    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tjd\" (UniqueName: \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.811294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.811120    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.811294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.811239    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.811688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.811658    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.812384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.812356    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.844101    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.844195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.841364    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3bf5d2\"/\"default-dockercfg-fjdhs\""
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.812813    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.844738    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.845348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.843010    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.847570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.847538    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tjd\" (UniqueName: \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:25.849902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:25.848880    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\") pod \"test-model-3bf5d2-kserve-78cf4559c9-s4gd9\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") " pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:26.092951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:26.090273    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:26.249421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:26.249384    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:26.249619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:26.249445    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:26.548595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:26.545805    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:19:27.037857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.018950    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:19:27.037857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.019038    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:19:27.249431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.249392    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:27.249616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.249453    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:27.271018 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.270975    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" event={"ID":"37762420-040d-446c-83b6-7d1e835bd5a3","Type":"ContainerStarted","Data":"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"}
Oct 30 15:19:27.271195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.271029    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" event={"ID":"37762420-040d-446c-83b6-7d1e835bd5a3","Type":"ContainerStarted","Data":"ab588a5f14d41962855858145ddd17d152177812df70d6991589c6baf6e8a7a7"}
Oct 30 15:19:27.332004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:27.319109    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" podStartSLOduration=2.319088847 podStartE2EDuration="2.319088847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:27.317786567 +0000 UTC m=+1916.759677297" watchObservedRunningTime="2026-10-30 15:19:27.319088847 +0000 UTC m=+1916.760979567"
Oct 30 15:19:28.091779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:28.091739    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:28.248706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:28.248672    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:28.248921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:28.248737    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:28.441879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:28.441841    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:28.455175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:28.455146    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:29.250305 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:29.250264    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 15:19:29.250939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:29.250903    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 15:19:29.327310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:29.327269    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:19:29.633602 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:19:29.633563    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:19:30.254005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.253973    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:30.254446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.254409    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:30.255345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.255323    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:19:30.314013 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.313977    5064 generic.go:356] "Generic (PLEG): container finished" podID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerID="0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db" exitCode=0
Oct 30 15:19:30.314237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.314053    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerDied","Data":"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"}
Oct 30 15:19:30.314590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.314570    5064 scope.go:122] "RemoveContainer" containerID="0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"
Oct 30 15:19:30.316667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.316634    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:30.316792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.316708    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:30.316865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:30.316765    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:31.348929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:31.348894    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerStarted","Data":"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"}
Oct 30 15:19:31.397716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:31.397681    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:19:31.397894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:31.397880    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:31.397941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:31.397922    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:32.398003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:32.397970    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:32.398418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:32.398023    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:32.454937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:32.454729    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:19:32.455132 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:32.455005    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerName="maas-api" containerID="cri-o://b897bfca622ab6806be6fa7046fdd73e6a616c3628004fb623324773fc75a123" gracePeriod=30
Oct 30 15:19:32.567981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:32.567144    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.139:8443/health\": dial tcp 10.132.0.139:8443: connect: connection refused"
Oct 30 15:19:33.371984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.371950    5064 generic.go:356] "Generic (PLEG): container finished" podID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerID="b897bfca622ab6806be6fa7046fdd73e6a616c3628004fb623324773fc75a123" exitCode=0
Oct 30 15:19:33.372164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.372063    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" event={"ID":"c45f4ed1-abf4-4068-a2d9-264b9b9d8125","Type":"ContainerDied","Data":"b897bfca622ab6806be6fa7046fdd73e6a616c3628004fb623324773fc75a123"}
Oct 30 15:19:33.399255 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.399221    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:33.399707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.399278    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:33.447040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.446749    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:19:33.548977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.548935    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\") pod \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") "
Oct 30 15:19:33.549242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.549213    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\") pod \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") "
Oct 30 15:19:33.549433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.549295    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\" (UniqueName: \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\") pod \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") "
Oct 30 15:19:33.549433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.549405    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\") pod \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\" (UID: \"c45f4ed1-abf4-4068-a2d9-264b9b9d8125\") "
Oct 30 15:19:33.551534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.551508    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca" pod "c45f4ed1-abf4-4068-a2d9-264b9b9d8125" (UID: "c45f4ed1-abf4-4068-a2d9-264b9b9d8125"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:19:33.552969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.552930    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln" pod "c45f4ed1-abf4-4068-a2d9-264b9b9d8125" (UID: "c45f4ed1-abf4-4068-a2d9-264b9b9d8125"). InnerVolumeSpecName "kube-api-access-sdkln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:19:33.553241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.553218    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs" pod "c45f4ed1-abf4-4068-a2d9-264b9b9d8125" (UID: "c45f4ed1-abf4-4068-a2d9-264b9b9d8125"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:19:33.554692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.554671    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls" pod "c45f4ed1-abf4-4068-a2d9-264b9b9d8125" (UID: "c45f4ed1-abf4-4068-a2d9-264b9b9d8125"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:19:33.652104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.652011    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:19:33.652104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.652048    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:19:33.652104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.652061    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkln\" (UniqueName: \"kubernetes.io/projected/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-kube-api-access-sdkln\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:19:33.652104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:33.652072    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45f4ed1-abf4-4068-a2d9-264b9b9d8125-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:19:34.381014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.380985    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"
Oct 30 15:19:34.381203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.380990    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx" event={"ID":"c45f4ed1-abf4-4068-a2d9-264b9b9d8125","Type":"ContainerDied","Data":"e19f52518d71a2440e27d7b6ddf24e3127f321e0c1a59b95f2dadf1f439d27a0"}
Oct 30 15:19:34.381203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.381110    5064 scope.go:122] "RemoveContainer" containerID="b897bfca622ab6806be6fa7046fdd73e6a616c3628004fb623324773fc75a123"
Oct 30 15:19:34.397739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.397716    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:34.397877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.397762    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:34.466193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.466150    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:19:34.472960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.472922    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-496acb-778648cc76-nd8cx"]
Oct 30 15:19:34.748985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:34.748897    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" path="/var/lib/kubelet/pods/c45f4ed1-abf4-4068-a2d9-264b9b9d8125/volumes"
Oct 30 15:19:35.397935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:35.397880    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:35.398242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:35.398211    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:36.093884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:36.092914    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:36.107864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:36.107818    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:36.401517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:36.401471    5064 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 15:19:36.401868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:36.401798    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 15:19:36.485368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:36.485334    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:19:37.404174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:37.404145    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:19:37.404638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:37.404618    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:19:37.405306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:37.405284    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:19:43.008108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.008023    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:19:43.009298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.009271    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerName="maas-api"
Oct 30 15:19:43.009523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.009497    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerName="maas-api"
Oct 30 15:19:43.009758 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.009741    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45f4ed1-abf4-4068-a2d9-264b9b9d8125" containerName="maas-api"
Oct 30 15:19:43.013579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.013560    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.040504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.040458    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:19:43.044998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.044961    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-859a8d-openshift-default-dockercfg-df5dq\""
Oct 30 15:19:43.092548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.092511    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.092757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.092670    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.092757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.092742    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.092934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.092906    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8fx\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.093077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.093058    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.093219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.093195    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.093275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.093235    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.093330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.093274    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.093387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.093352    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198130    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198254    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8fx\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198383    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198487    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198526    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198572    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198640    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198758    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.198903    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.203848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.199526    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.216852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.208890    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.216852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.214931    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.216852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.215225    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.216852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.215486    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.216852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.216165    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.222948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.222911    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.269051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.265861    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.274766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.274190    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8fx\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\") pod \"e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") " pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.340196 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.340155    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:43.823534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.823498    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:19:43.826514 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:19:43.825937    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18362fdf_512b_4462_81bb_f3733400fb17.slice/crio-8f0e363780e64bc8458dd199f83779c9348125d3baad5e1193cd0e20080db8ce WatchSource:0}: Error finding container 8f0e363780e64bc8458dd199f83779c9348125d3baad5e1193cd0e20080db8ce: Status 404 returned error can't find the container with id 8f0e363780e64bc8458dd199f83779c9348125d3baad5e1193cd0e20080db8ce
Oct 30 15:19:43.830291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.830222    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:43.830419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.830377    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:43.830488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:43.830424    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:19:44.530469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:44.530425    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerStarted","Data":"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"}
Oct 30 15:19:44.530931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:44.530494    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerStarted","Data":"8f0e363780e64bc8458dd199f83779c9348125d3baad5e1193cd0e20080db8ce"}
Oct 30 15:19:44.595767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:44.595714    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podStartSLOduration=2.595700979 podStartE2EDuration="2.595700979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:44.592723709 +0000 UTC m=+1934.034614425" watchObservedRunningTime="2026-10-30 15:19:44.595700979 +0000 UTC m=+1934.037591696"
Oct 30 15:19:45.341766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:45.341731    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:45.347934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:45.347909    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:45.539696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:45.539662    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:45.540752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:19:45.540728    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:19:59.632905 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:19:59.632816    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:20:29.650005 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:20:29.649955    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:20:59.633198 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:20:59.633151    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:21:05.040740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:05.040693    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:21:09.602605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:09.602566    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:21:09.603042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:09.602858    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy" containerID="cri-o://9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e" gracePeriod=30
Oct 30 15:21:14.902404 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:14.902371    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:21:15.058691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058653    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.058909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058726    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.058909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058788    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.058909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058849    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058911    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.058963    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059001    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059027    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:15.059315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059203    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059247    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\") pod \"b194ba01-7348-4500-a555-b2009e03d314\" (UID: \"b194ba01-7348-4500-a555-b2009e03d314\") "
Oct 30 15:21:15.059315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059272    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:15.059486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059305    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:21:15.059578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059553    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:15.059741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.059698    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:15.060213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.060191    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.060267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.060221    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.060267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.060237    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.060267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.060251    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.060363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.060265    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b194ba01-7348-4500-a555-b2009e03d314-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.061256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.061230    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:15.061368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.061350    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "kube-api-access-chhg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:21:15.061637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.061611    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:21:15.061723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.061633    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token" pod "b194ba01-7348-4500-a555-b2009e03d314" (UID: "b194ba01-7348-4500-a555-b2009e03d314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:21:15.162104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.161991    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chhg4\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-kube-api-access-chhg4\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.162104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.162023    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b194ba01-7348-4500-a555-b2009e03d314-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.162104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.162033    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b194ba01-7348-4500-a555-b2009e03d314-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.162104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.162041    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b194ba01-7348-4500-a555-b2009e03d314-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:15.753855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.739631    5064 generic.go:356] "Generic (PLEG): container finished" podID="b194ba01-7348-4500-a555-b2009e03d314" containerID="9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e" exitCode=0
Oct 30 15:21:15.753855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.739821    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"
Oct 30 15:21:15.753855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.740102    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" event={"ID":"b194ba01-7348-4500-a555-b2009e03d314","Type":"ContainerDied","Data":"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"}
Oct 30 15:21:15.753855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.740279    5064 scope.go:122] "RemoveContainer" containerID="9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"
Oct 30 15:21:15.753855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.740151    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv" event={"ID":"b194ba01-7348-4500-a555-b2009e03d314","Type":"ContainerDied","Data":"4fcbf10729453cd619262cd9892ad523757561f444bff5191618610e682d807d"}
Oct 30 15:21:15.821857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.814273    5064 scope.go:122] "RemoveContainer" containerID="9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"
Oct 30 15:21:15.821857 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:21:15.814855    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e\": container with ID starting with 9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e not found: ID does not exist" containerID="9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"
Oct 30 15:21:15.821857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.814896    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e"} err="failed to get container status \"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e\": rpc error: code = NotFound desc = could not find container \"9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e\": container with ID starting with 9129ab814372ca8e247bc73c6c588e8f7a972022d582c54ec0f265beac09000e not found: ID does not exist"
Oct 30 15:21:15.848849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.846959    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:21:15.858886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:15.858801    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ssmkv"]
Oct 30 15:21:16.728738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:16.728703    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b194ba01-7348-4500-a555-b2009e03d314" path="/var/lib/kubelet/pods/b194ba01-7348-4500-a555-b2009e03d314/volumes"
Oct 30 15:21:20.069121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.069074    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" podUID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" containerName="main" containerID="cri-o://939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f" gracePeriod=45
Oct 30 15:21:20.395558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.395522    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:21:20.554625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554590    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.554838 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554707    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.554838 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554789    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.554838 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554820    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\" (UniqueName: \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.555021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554872    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\" (UniqueName: \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.555021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.554916    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:20.555137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555047    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\") pod \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\" (UID: \"365e40cd-e16f-4df8-a5fa-dcb7dcafd352\") "
Oct 30 15:21:20.555137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555041    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:20.555254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555088    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:20.555863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555811    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.556011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555871    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.556011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.555881    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.557293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.557259    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:21:20.557293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.557272    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:21:20.557426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.557363    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789" pod "365e40cd-e16f-4df8-a5fa-dcb7dcafd352" (UID: "365e40cd-e16f-4df8-a5fa-dcb7dcafd352"). InnerVolumeSpecName "kube-api-access-8l789". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:21:20.657496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.657455    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.657496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.657485    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l789\" (UniqueName: \"kubernetes.io/projected/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-kube-api-access-8l789\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.657496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.657494    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365e40cd-e16f-4df8-a5fa-dcb7dcafd352-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:21:20.848598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.848562    5064 generic.go:356] "Generic (PLEG): container finished" podID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" containerID="939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f" exitCode=0
Oct 30 15:21:20.848772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.848620    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"
Oct 30 15:21:20.848772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.848649    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" event={"ID":"365e40cd-e16f-4df8-a5fa-dcb7dcafd352","Type":"ContainerDied","Data":"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"}
Oct 30 15:21:20.848772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.848704    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht" event={"ID":"365e40cd-e16f-4df8-a5fa-dcb7dcafd352","Type":"ContainerDied","Data":"4d2c6a24a943ec9c4527e8ceba77c5870e616491b9c4483c032db8425ddbc8ab"}
Oct 30 15:21:20.848772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.848730    5064 scope.go:122] "RemoveContainer" containerID="939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"
Oct 30 15:21:20.859935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.859914    5064 scope.go:122] "RemoveContainer" containerID="939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"
Oct 30 15:21:20.860255 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:21:20.860232    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f\": container with ID starting with 939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f not found: ID does not exist" containerID="939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"
Oct 30 15:21:20.860308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.860265    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f"} err="failed to get container status \"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f\": rpc error: code = NotFound desc = could not find container \"939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f\": container with ID starting with 939817fc272b36c421cbd1884190b2473ed84e3390031b8287eab13170d55d2f not found: ID does not exist"
Oct 30 15:21:20.907862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.907743    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:21:20.911915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:20.911885    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/auth-test-model-100b2e-kserve-5bccd76f65-fjbht"]
Oct 30 15:21:22.725934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:22.725890    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" path="/var/lib/kubelet/pods/365e40cd-e16f-4df8-a5fa-dcb7dcafd352/volumes"
Oct 30 15:21:29.636526 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:21:29.636479    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:21:31.583021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:31.582980    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:21:31.583448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:31.583087    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:21:34.077866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.077769    5064 generic.go:356] "Generic (PLEG): container finished" podID="18362fdf-512b-4462-81bb-f3733400fb17" containerID="c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081" exitCode=0
Oct 30 15:21:34.078300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.077854    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerDied","Data":"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"}
Oct 30 15:21:34.078343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.078308    5064 scope.go:122] "RemoveContainer" containerID="c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"
Oct 30 15:21:34.080251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.080215    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:21:34.080370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.080290    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:21:34.080370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:34.080327    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:21:35.087727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:35.087692    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerStarted","Data":"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"}
Oct 30 15:21:35.344269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:35.344184    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:21:35.344741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:35.344627    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:35.344978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:35.344952    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:36.356014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:36.349943    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:36.356014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:36.350016    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:37.356855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:37.347978    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:37.356855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:37.348043    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:38.342240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:38.342201    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:38.342448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:38.342264    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:39.354204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:39.354166    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:39.354930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:39.354898    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:40.342016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:40.341974    5064 patch_prober.go:29] interesting pod/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 15:21:40.342245 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:40.342039    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 15:21:41.401859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:41.392705    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:21:41.401859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:41.394476    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:21:41.402792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:21:41.402759    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:21:59.632731 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:21:59.632694    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:22:10.516536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:10.516499    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:22:19.423951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.423910    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:22:19.424972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.424942    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy"
Oct 30 15:22:19.424972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.424971    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy"
Oct 30 15:22:19.425174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.425024    5064 cpu_manager.go:414] "containerMap: removing container" podUID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" containerName="main"
Oct 30 15:22:19.425174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.425033    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" containerName="main"
Oct 30 15:22:19.425249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.425218    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b194ba01-7348-4500-a555-b2009e03d314" containerName="istio-proxy"
Oct 30 15:22:19.425249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.425234    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="365e40cd-e16f-4df8-a5fa-dcb7dcafd352" containerName="main"
Oct 30 15:22:19.428297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.428278    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.448604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.448575    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:22:19.464052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.464020    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-13009a-serving-cert\""
Oct 30 15:22:19.534329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.534289    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krj84\" (UniqueName: \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.534519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.534435    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.534519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.534508    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.534673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.534650    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.640437    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krj84\" (UniqueName: \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.640557    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.640635    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.640710    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:22:19.641005    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-serving-cert: secret "maas-api-e2e-inf-b-13009a-serving-cert" not found
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:22:19.641082    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls podName:486820ad-d290-4ec4-aebc-ebaf2fc41112 nodeName:}" failed. No retries permitted until 2026-10-30 15:22:20.141059942 +0000 UTC m=+2089.582950642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls") pod "maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" (UID: "486820ad-d290-4ec4-aebc-ebaf2fc41112") : secret "maas-api-e2e-inf-b-13009a-serving-cert" not found
Oct 30 15:22:19.643856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.642341    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.652900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.652794    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:19.682910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:19.678725    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krj84\" (UniqueName: \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:20.151328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:20.151294    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:20.154197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:20.154148    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") pod \"maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:20.342131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:20.342097    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:20.818016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:20.817981    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:22:20.880461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:20.880405    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:22:21.595232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:21.595123    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" event={"ID":"486820ad-d290-4ec4-aebc-ebaf2fc41112","Type":"ContainerStarted","Data":"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"}
Oct 30 15:22:21.595232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:21.595184    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" event={"ID":"486820ad-d290-4ec4-aebc-ebaf2fc41112","Type":"ContainerStarted","Data":"2d3c2c8fbf4038f33c06e636742660edafc870f7be02b5f1d4c16909318a06e4"}
Oct 30 15:22:21.595232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:21.595214    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:21.643345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:21.643293    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" podStartSLOduration=2.191804077 podStartE2EDuration="2.643279228s" totalImagesPullingTime="451.475151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:21.640104732 +0000 UTC m=+2091.081995448" watchObservedRunningTime="2026-10-30 15:22:21.643279228 +0000 UTC m=+2091.085169970"
Oct 30 15:22:25.541602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.541562    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" podUID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" containerName="main" containerID="cri-o://548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd" gracePeriod=45
Oct 30 15:22:25.849751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.849721    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:22:25.942134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942100    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942165    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\" (UniqueName: \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942362    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942441    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942563    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942643    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\") pod \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\" (UID: \"1a5a168d-a066-4694-8225-a3f7fe95eb8f\") "
Oct 30 15:22:25.942918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.942433    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:22:25.943252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.943182    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:22:25.943252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.943213    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:22:25.943606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.943585    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:25.943700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.943609    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:25.943700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.943621    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:25.945213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.945191    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "kube-api-access-hxsbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:22:25.945486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.945465    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:22:25.945554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:25.945528    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm" pod "1a5a168d-a066-4694-8225-a3f7fe95eb8f" (UID: "1a5a168d-a066-4694-8225-a3f7fe95eb8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:22:26.045316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.045281    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsbl\" (UniqueName: \"kubernetes.io/projected/1a5a168d-a066-4694-8225-a3f7fe95eb8f-kube-api-access-hxsbl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:26.045316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.045309    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5a168d-a066-4694-8225-a3f7fe95eb8f-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:26.045316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.045319    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5a168d-a066-4694-8225-a3f7fe95eb8f-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:26.659855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.658193    5064 generic.go:356] "Generic (PLEG): container finished" podID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" containerID="548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd" exitCode=0
Oct 30 15:22:26.659855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.658537    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" event={"ID":"1a5a168d-a066-4694-8225-a3f7fe95eb8f","Type":"ContainerDied","Data":"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"}
Oct 30 15:22:26.659855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.658586    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx" event={"ID":"1a5a168d-a066-4694-8225-a3f7fe95eb8f","Type":"ContainerDied","Data":"b21e45df693cc6c05518bc8281b2cd0f14ed32da7945bc4d39bbe007fc11b026"}
Oct 30 15:22:26.659855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.658608    5064 scope.go:122] "RemoveContainer" containerID="548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"
Oct 30 15:22:26.659855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.658866    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"
Oct 30 15:22:26.716181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.716150    5064 scope.go:122] "RemoveContainer" containerID="548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"
Oct 30 15:22:26.716581 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:22:26.716550    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd\": container with ID starting with 548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd not found: ID does not exist" containerID="548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"
Oct 30 15:22:26.716672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.716595    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd"} err="failed to get container status \"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd\": rpc error: code = NotFound desc = could not find container \"548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd\": container with ID starting with 548781323fe62436e2c6fa6dffa313e2c74e57dc0ab87bce264dafa1e8133fdd not found: ID does not exist"
Oct 30 15:22:26.754292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.754255    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:22:26.758755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:26.758726    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-765228/test-model-765228-kserve-7ccf64cb99-xbcrx"]
Oct 30 15:22:27.609338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:27.609307    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:22:28.726082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:28.726049    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" path="/var/lib/kubelet/pods/1a5a168d-a066-4694-8225-a3f7fe95eb8f/volumes"
Oct 30 15:22:29.632087 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:22:29.632049    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:22:31.158820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:31.158787    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:22:31.159754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:31.159730    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:22:32.185171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.185128    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:22:32.187108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.187084    5064 cpu_manager.go:414] "containerMap: removing container" podUID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" containerName="main"
Oct 30 15:22:32.187283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.187270    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" containerName="main"
Oct 30 15:22:32.187655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.187638    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5a168d-a066-4694-8225-a3f7fe95eb8f" containerName="main"
Oct 30 15:22:32.234306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.234266    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:22:32.234696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.234606    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.253660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.253576    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d84bc6-openshift-default-dockercfg-xc7z7\""
Oct 30 15:22:32.304274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.304242    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.304521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.304496    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.304786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.304762    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.305073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.305050    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlndc\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.305296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.305273    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.305575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.305554    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.305860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.305822    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.306020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.306002    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.306184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.306164    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.407763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.407724    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.407902    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlndc\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.407945    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.407975    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408139    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408188    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408206    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408278    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408353    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408386    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408589    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408710    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408730    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.408789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.408750    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.410410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.410388    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.410659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.410639    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.433652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.433616    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.433821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.433752    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlndc\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.580714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.580620    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:32.907504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.907474    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:22:32.907610 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:22:32.907484    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f3acbd_0587_4145_805f_a0b142364dd4.slice/crio-1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c WatchSource:0}: Error finding container 1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c: Status 404 returned error can't find the container with id 1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c
Oct 30 15:22:32.910724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.910683    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:22:32.910858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.910770    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:22:32.910933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:32.910815    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:22:33.732374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:33.732331    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" event={"ID":"62f3acbd-0587-4145-805f-a0b142364dd4","Type":"ContainerStarted","Data":"7400023a7dd4894a5857a67bffe0c600d357702ef75fe92012256ed7ee563698"}
Oct 30 15:22:33.732812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:33.732396    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" event={"ID":"62f3acbd-0587-4145-805f-a0b142364dd4","Type":"ContainerStarted","Data":"1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c"}
Oct 30 15:22:33.790817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:33.790751    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podStartSLOduration=1.7907319560000001 podStartE2EDuration="1.790731956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:33.787334133 +0000 UTC m=+2103.229224878" watchObservedRunningTime="2026-10-30 15:22:33.790731956 +0000 UTC m=+2103.232622674"
Oct 30 15:22:34.581741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:34.581699    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:34.583222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:34.583196    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:34.583300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:34.583249    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:35.582086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:35.582040    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:35.582496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:35.582116    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:36.581939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:36.581905    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:36.582115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:36.581994    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:37.581969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:37.581925    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:37.582200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:37.582009    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:38.590823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:38.588273    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:38.590823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:38.588338    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:39.581236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:39.581196    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 15:22:39.581441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:39.581264    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 15:22:40.586598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:40.586568    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:40.587127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:40.586940    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:40.587496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:40.587474    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:22:51.142739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:51.142705    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:22:55.541922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:55.541882    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:22:55.542498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:55.542137    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" podUID="ced65a45-9933-453e-a2a1-846a3936d141" containerName="maas-api" containerID="cri-o://55d2f2afc4acf4f40ccfa415438ff951165b8104c1cb27c4003d95e77ed3c9ac" gracePeriod=30
Oct 30 15:22:56.058378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.058339    5064 generic.go:356] "Generic (PLEG): container finished" podID="ced65a45-9933-453e-a2a1-846a3936d141" containerID="55d2f2afc4acf4f40ccfa415438ff951165b8104c1cb27c4003d95e77ed3c9ac" exitCode=0
Oct 30 15:22:56.058913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.058884    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" event={"ID":"ced65a45-9933-453e-a2a1-846a3936d141","Type":"ContainerDied","Data":"55d2f2afc4acf4f40ccfa415438ff951165b8104c1cb27c4003d95e77ed3c9ac"}
Oct 30 15:22:56.059116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.059098    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9" event={"ID":"ced65a45-9933-453e-a2a1-846a3936d141","Type":"ContainerDied","Data":"0ed2e3ac36e61dde17c524cc97f27150eb2c1d7b4f1849e0c4ebb180f13ed24a"}
Oct 30 15:22:56.059225 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.059212    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed2e3ac36e61dde17c524cc97f27150eb2c1d7b4f1849e0c4ebb180f13ed24a"
Oct 30 15:22:56.060557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.060528    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:22:56.075658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.075571    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:22:56.155376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.148930    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\") pod \"ced65a45-9933-453e-a2a1-846a3936d141\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") "
Oct 30 15:22:56.155376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.149056    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\") pod \"ced65a45-9933-453e-a2a1-846a3936d141\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") "
Oct 30 15:22:56.155376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.149101    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\") pod \"ced65a45-9933-453e-a2a1-846a3936d141\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") "
Oct 30 15:22:56.155376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.149228    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\") pod \"ced65a45-9933-453e-a2a1-846a3936d141\" (UID: \"ced65a45-9933-453e-a2a1-846a3936d141\") "
Oct 30 15:22:56.155376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.150313    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca" pod "ced65a45-9933-453e-a2a1-846a3936d141" (UID: "ced65a45-9933-453e-a2a1-846a3936d141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:22:56.209904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.181868    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced65a45-9933-453e-a2a1-846a3936d141-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:56.431058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.430996    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls" pod "ced65a45-9933-453e-a2a1-846a3936d141" (UID: "ced65a45-9933-453e-a2a1-846a3936d141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:22:56.435056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.435003    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs" pod "ced65a45-9933-453e-a2a1-846a3936d141" (UID: "ced65a45-9933-453e-a2a1-846a3936d141"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:22:56.437474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.437429    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt" pod "ced65a45-9933-453e-a2a1-846a3936d141" (UID: "ced65a45-9933-453e-a2a1-846a3936d141"). InnerVolumeSpecName "kube-api-access-7szpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:22:56.501710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.501619    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:56.501710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.501653    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/ced65a45-9933-453e-a2a1-846a3936d141-kube-api-access-7szpt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:56.501710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:56.501664    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ced65a45-9933-453e-a2a1-846a3936d141-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:22:57.066192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:57.066151    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"
Oct 30 15:22:57.133784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:57.133744    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:22:57.145211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:57.145181    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a65941-5fb4d66b55-mtmg9"]
Oct 30 15:22:58.751202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:22:58.751162    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced65a45-9933-453e-a2a1-846a3936d141" path="/var/lib/kubelet/pods/ced65a45-9933-453e-a2a1-846a3936d141/volumes"
Oct 30 15:22:59.632775 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:22:59.632734    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:23:06.167472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.167411    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" containerName="main" containerID="cri-o://818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e" gracePeriod=45
Oct 30 15:23:06.551405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.551373    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:23:06.588278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588243    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\" (UniqueName: \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.588454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588412    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.588572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588557    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.588648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588632    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.589625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.589625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588896    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\") pod \"37762420-040d-446c-83b6-7d1e835bd5a3\" (UID: \"37762420-040d-446c-83b6-7d1e835bd5a3\") "
Oct 30 15:23:06.589625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.588954    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:06.589625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.589207    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:06.589625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.589593    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:06.590011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.589651    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:06.590011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.589667    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:06.590909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.590821    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:06.591025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.590908    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:06.591025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.590990    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd" pod "37762420-040d-446c-83b6-7d1e835bd5a3" (UID: "37762420-040d-446c-83b6-7d1e835bd5a3"). InnerVolumeSpecName "kube-api-access-x7tjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:06.691100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.691063    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37762420-040d-446c-83b6-7d1e835bd5a3-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:06.691100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.691097    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:06.691100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.691106    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tjd\" (UniqueName: \"kubernetes.io/projected/37762420-040d-446c-83b6-7d1e835bd5a3-kube-api-access-x7tjd\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:06.691357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:06.691115    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37762420-040d-446c-83b6-7d1e835bd5a3-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:07.195528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.195481    5064 generic.go:356] "Generic (PLEG): container finished" podID="37762420-040d-446c-83b6-7d1e835bd5a3" containerID="818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e" exitCode=0
Oct 30 15:23:07.196590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.195570    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"
Oct 30 15:23:07.196590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.195570    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" event={"ID":"37762420-040d-446c-83b6-7d1e835bd5a3","Type":"ContainerDied","Data":"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"}
Oct 30 15:23:07.196590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.195624    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" event={"ID":"37762420-040d-446c-83b6-7d1e835bd5a3","Type":"ContainerDied","Data":"ab588a5f14d41962855858145ddd17d152177812df70d6991589c6baf6e8a7a7"}
Oct 30 15:23:07.196590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.195655    5064 scope.go:122] "RemoveContainer" containerID="818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"
Oct 30 15:23:07.208114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.208082    5064 scope.go:122] "RemoveContainer" containerID="818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"
Oct 30 15:23:07.208566 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:07.208522    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e\": container with ID starting with 818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e not found: ID does not exist" containerID="818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"
Oct 30 15:23:07.208688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.208593    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e"} err="failed to get container status \"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e\": rpc error: code = NotFound desc = could not find container \"818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e\": container with ID starting with 818c833cc97851086da062e051b1b7af92cc70a201b80c7eae765bc655652c3e not found: ID does not exist"
Oct 30 15:23:07.256172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.256134    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:23:07.269945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.269906    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9"]
Oct 30 15:23:07.483504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:07.483392    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3bf5d2/test-model-3bf5d2-kserve-78cf4559c9-s4gd9" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:23:08.725047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:08.725015    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" path="/var/lib/kubelet/pods/37762420-040d-446c-83b6-7d1e835bd5a3/volumes"
Oct 30 15:23:11.099935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.099888    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerName="main" containerID="cri-o://ff1d4cf1a015b604f25a496eac4f576d81a2d96191ff4d95456c61dc4660f15a" gracePeriod=45
Oct 30 15:23:11.231240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.231212    5064 generic.go:356] "Generic (PLEG): container finished" podID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerID="ff1d4cf1a015b604f25a496eac4f576d81a2d96191ff4d95456c61dc4660f15a" exitCode=0
Oct 30 15:23:11.231382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.231287    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" event={"ID":"0a93b058-4eed-4e3d-b397-192ea837fec6","Type":"ContainerDied","Data":"ff1d4cf1a015b604f25a496eac4f576d81a2d96191ff4d95456c61dc4660f15a"}
Oct 30 15:23:11.397193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.397156    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:23:11.513153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513110    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.513337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513235    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.513421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513382    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513507    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513541    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513706    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\" (UniqueName: \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\") pod \"0a93b058-4eed-4e3d-b397-192ea837fec6\" (UID: \"0a93b058-4eed-4e3d-b397-192ea837fec6\") "
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.513762    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.514037    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.514722    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.514744    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:11.515324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.515209    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:11.516195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.516168    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:11.516501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.516471    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:11.519366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.519336    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf" pod "0a93b058-4eed-4e3d-b397-192ea837fec6" (UID: "0a93b058-4eed-4e3d-b397-192ea837fec6"). InnerVolumeSpecName "kube-api-access-xdtgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:11.626851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.623638    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:11.626851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.623679    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdtgf\" (UniqueName: \"kubernetes.io/projected/0a93b058-4eed-4e3d-b397-192ea837fec6-kube-api-access-xdtgf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:11.626851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.623698    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a93b058-4eed-4e3d-b397-192ea837fec6-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:11.626851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:11.623711    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a93b058-4eed-4e3d-b397-192ea837fec6-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:12.240679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.240651    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"
Oct 30 15:23:12.241236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.240686    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" event={"ID":"0a93b058-4eed-4e3d-b397-192ea837fec6","Type":"ContainerDied","Data":"2248534ca10c203591b9129935ea29da6097360f8c9375e19f7b0a93c596b5e7"}
Oct 30 15:23:12.241236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.240740    5064 scope.go:122] "RemoveContainer" containerID="ff1d4cf1a015b604f25a496eac4f576d81a2d96191ff4d95456c61dc4660f15a"
Oct 30 15:23:12.302540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.302504    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:23:12.303637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.303607    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.149:8000/ready\": context deadline exceeded"
Oct 30 15:23:12.316029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.315983    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-34218d/test-model-34218d-kserve-67cd556df9-smhkv"]
Oct 30 15:23:12.725416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:12.725380    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" path="/var/lib/kubelet/pods/0a93b058-4eed-4e3d-b397-192ea837fec6/volumes"
Oct 30 15:23:27.875035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:27.874994    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:23:27.875634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:27.875234    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy" containerID="cri-o://426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec" gracePeriod=30
Oct 30 15:23:28.465992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:28.465958    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:23:28.466186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:28.466162    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" containerID="cri-o://134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0" gracePeriod=30
Oct 30 15:23:29.654801 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:29.654756    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:23:32.140473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:32.140429    5064 scope.go:122] "RemoveContainer" containerID="55d2f2afc4acf4f40ccfa415438ff951165b8104c1cb27c4003d95e77ed3c9ac"
Oct 30 15:23:33.178667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.178640    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:23:33.319194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319161    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319238    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319361    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319425    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319510    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319526    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.319673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319583    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319627    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319781 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319715    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319783    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\") pod \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\" (UID: \"942d2e88-2f2e-48b9-bf7c-11b0402667e2\") "
Oct 30 15:23:33.319863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319794    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.320317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.319965    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.320317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320039    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.320317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320275    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:33.320645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320623    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.320713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320651    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.320713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320666    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.320713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320681    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.320713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.320694    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.322306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.322277    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:33.322701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.322666    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "kube-api-access-7p7jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:33.322841 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.322725    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:23:33.322987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.322962    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy" pod "942d2e88-2f2e-48b9-bf7c-11b0402667e2" (UID: "942d2e88-2f2e-48b9-bf7c-11b0402667e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.441911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.435646    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.441911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.435694    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7jx\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-kube-api-access-7p7jx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.441911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.435707    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.441911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.435721    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942d2e88-2f2e-48b9-bf7c-11b0402667e2-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.456966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.456921    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:23:33.457287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.457252    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:23:33.570131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.570096    5064 generic.go:356] "Generic (PLEG): container finished" podID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerID="426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec" exitCode=0
Oct 30 15:23:33.570318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.570167    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" event={"ID":"942d2e88-2f2e-48b9-bf7c-11b0402667e2","Type":"ContainerDied","Data":"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"}
Oct 30 15:23:33.570318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.570221    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz" event={"ID":"942d2e88-2f2e-48b9-bf7c-11b0402667e2","Type":"ContainerDied","Data":"8c7080e52ab1f8bfe92a10bd7695dfc9851436b8bd525f6edd9f5141d6d80c18"}
Oct 30 15:23:33.570318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.570249    5064 scope.go:122] "RemoveContainer" containerID="426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"
Oct 30 15:23:33.570318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.570167    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"
Oct 30 15:23:33.584053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.584032    5064 scope.go:122] "RemoveContainer" containerID="426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"
Oct 30 15:23:33.584396 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:33.584373    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec\": container with ID starting with 426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec not found: ID does not exist" containerID="426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"
Oct 30 15:23:33.584474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.584407    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec"} err="failed to get container status \"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec\": rpc error: code = NotFound desc = could not find container \"426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec\": container with ID starting with 426e78c2e8a3d606ac7d4f2f266f4a207904a91cf1f5fecf390f285a9adfcdec not found: ID does not exist"
Oct 30 15:23:33.630941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.630904    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:23:33.634665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.634629    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-kdlpz"]
Oct 30 15:23:33.732636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.732612    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:23:33.739024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739000    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739117    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739218    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739252    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739278    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.739384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739365    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739497    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739543    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739604    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739560    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:33.739659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739637    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\") pod \"3be869ab-5708-4180-80f7-801d1a44ec28\" (UID: \"3be869ab-5708-4180-80f7-801d1a44ec28\") "
Oct 30 15:23:33.739659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.739905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739751    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.739905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.739816    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.740427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.740402    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.740530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.740430    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.740530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.740445    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.740530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.740459    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be869ab-5708-4180-80f7-801d1a44ec28-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.740530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.740474    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.741395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.741367    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:33.741566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.741547    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:33.741610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.741580    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:23:33.741737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.741717    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h" pod "3be869ab-5708-4180-80f7-801d1a44ec28" (UID: "3be869ab-5708-4180-80f7-801d1a44ec28"). InnerVolumeSpecName "kube-api-access-f5x9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:33.842182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.842142    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be869ab-5708-4180-80f7-801d1a44ec28-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.842182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.842180    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.842466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.842195    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5x9h\" (UniqueName: \"kubernetes.io/projected/3be869ab-5708-4180-80f7-801d1a44ec28-kube-api-access-f5x9h\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:33.842466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:33.842209    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be869ab-5708-4180-80f7-801d1a44ec28-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:34.580227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.580185    5064 generic.go:356] "Generic (PLEG): container finished" podID="3be869ab-5708-4180-80f7-801d1a44ec28" containerID="134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0" exitCode=0
Oct 30 15:23:34.580677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.580233    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" event={"ID":"3be869ab-5708-4180-80f7-801d1a44ec28","Type":"ContainerDied","Data":"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"}
Oct 30 15:23:34.580677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.580239    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"
Oct 30 15:23:34.580677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.580285    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh" event={"ID":"3be869ab-5708-4180-80f7-801d1a44ec28","Type":"ContainerDied","Data":"ddd91e8a38f0e8c7a1b6f311b518db69bfc0e0ef39d1884e82d49a3f64c69417"}
Oct 30 15:23:34.580677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.580315    5064 scope.go:122] "RemoveContainer" containerID="134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"
Oct 30 15:23:34.591681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.591658    5064 scope.go:122] "RemoveContainer" containerID="134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"
Oct 30 15:23:34.592023 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:34.592003    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0\": container with ID starting with 134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0 not found: ID does not exist" containerID="134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"
Oct 30 15:23:34.592087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.592035    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0"} err="failed to get container status \"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0\": rpc error: code = NotFound desc = could not find container \"134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0\": container with ID starting with 134bcd9f518ee830b75738d38c20ae3b359bd8a3dee8244a81988dd69783a0e0 not found: ID does not exist"
Oct 30 15:23:34.654430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.654395    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:23:34.667766 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.667723    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-99njh"]
Oct 30 15:23:34.724998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.724961    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" path="/var/lib/kubelet/pods/3be869ab-5708-4180-80f7-801d1a44ec28/volumes"
Oct 30 15:23:34.725641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:34.725618    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" path="/var/lib/kubelet/pods/942d2e88-2f2e-48b9-bf7c-11b0402667e2/volumes"
Oct 30 15:23:36.728045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:36.728009    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:23:36.728538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:36.728280    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" podUID="f55454ad-8f18-4afa-a3e5-b51944decbd2" containerName="maas-api" containerID="cri-o://5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049" gracePeriod=30
Oct 30 15:23:37.130884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.121548    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:23:37.279852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.250341    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\") pod \"f55454ad-8f18-4afa-a3e5-b51944decbd2\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") "
Oct 30 15:23:37.279852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.250501    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\" (UniqueName: \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\") pod \"f55454ad-8f18-4afa-a3e5-b51944decbd2\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") "
Oct 30 15:23:37.279852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.250633    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\") pod \"f55454ad-8f18-4afa-a3e5-b51944decbd2\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") "
Oct 30 15:23:37.279852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.250680    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\") pod \"f55454ad-8f18-4afa-a3e5-b51944decbd2\" (UID: \"f55454ad-8f18-4afa-a3e5-b51944decbd2\") "
Oct 30 15:23:37.398512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.395614    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca" pod "f55454ad-8f18-4afa-a3e5-b51944decbd2" (UID: "f55454ad-8f18-4afa-a3e5-b51944decbd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:37.431857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.424174    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt" pod "f55454ad-8f18-4afa-a3e5-b51944decbd2" (UID: "f55454ad-8f18-4afa-a3e5-b51944decbd2"). InnerVolumeSpecName "kube-api-access-2m9wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:37.431857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.428263    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls" pod "f55454ad-8f18-4afa-a3e5-b51944decbd2" (UID: "f55454ad-8f18-4afa-a3e5-b51944decbd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:37.448074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.439534    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs" pod "f55454ad-8f18-4afa-a3e5-b51944decbd2" (UID: "f55454ad-8f18-4afa-a3e5-b51944decbd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:37.448232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.448166    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:37.448232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.448198    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9wt\" (UniqueName: \"kubernetes.io/projected/f55454ad-8f18-4afa-a3e5-b51944decbd2-kube-api-access-2m9wt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:37.448232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.448212    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55454ad-8f18-4afa-a3e5-b51944decbd2-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:37.448232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.448224    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55454ad-8f18-4afa-a3e5-b51944decbd2-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:37.606950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.606913    5064 generic.go:356] "Generic (PLEG): container finished" podID="f55454ad-8f18-4afa-a3e5-b51944decbd2" containerID="5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049" exitCode=0
Oct 30 15:23:37.607162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.606976    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"
Oct 30 15:23:37.607162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.606997    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" event={"ID":"f55454ad-8f18-4afa-a3e5-b51944decbd2","Type":"ContainerDied","Data":"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"}
Oct 30 15:23:37.607162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.607055    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf" event={"ID":"f55454ad-8f18-4afa-a3e5-b51944decbd2","Type":"ContainerDied","Data":"6cfcef5324c3b8130f4c5d55e4f9166d534da669208c4e5ffa97c164a93cc9f3"}
Oct 30 15:23:37.607162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.607084    5064 scope.go:122] "RemoveContainer" containerID="5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"
Oct 30 15:23:37.619450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.619429    5064 scope.go:122] "RemoveContainer" containerID="5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"
Oct 30 15:23:37.619753 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:37.619734    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049\": container with ID starting with 5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049 not found: ID does not exist" containerID="5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"
Oct 30 15:23:37.619808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.619761    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049"} err="failed to get container status \"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049\": rpc error: code = NotFound desc = could not find container \"5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049\": container with ID starting with 5661b8cb1a784d5ca4a76f74f3ed1f9e61890941f60423cfe9b1d46b94b12049 not found: ID does not exist"
Oct 30 15:23:37.668514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.668425    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:23:37.682481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:37.682446    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34218d-7b8f977db7-68vzf"]
Oct 30 15:23:38.725561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:38.725527    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55454ad-8f18-4afa-a3e5-b51944decbd2" path="/var/lib/kubelet/pods/f55454ad-8f18-4afa-a3e5-b51944decbd2/volumes"
Oct 30 15:23:43.322135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:43.322099    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:23:43.322548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:43.322317    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy" containerID="cri-o://c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e" gracePeriod=30
Oct 30 15:23:45.110795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.110749    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:23:45.111406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.111118    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" containerID="cri-o://ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e" gracePeriod=30
Oct 30 15:23:45.379859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.373713    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:23:45.380099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.380066    5064 cpu_manager.go:414] "containerMap: removing container" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" containerName="main"
Oct 30 15:23:45.387926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.387895    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" containerName="main"
Oct 30 15:23:45.388127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388115    5064 cpu_manager.go:414] "containerMap: removing container" podUID="ced65a45-9933-453e-a2a1-846a3936d141" containerName="maas-api"
Oct 30 15:23:45.388217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388206    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced65a45-9933-453e-a2a1-846a3936d141" containerName="maas-api"
Oct 30 15:23:45.388313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388304    5064 cpu_manager.go:414] "containerMap: removing container" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy"
Oct 30 15:23:45.388376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388367    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy"
Oct 30 15:23:45.388512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388501    5064 cpu_manager.go:414] "containerMap: removing container" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy"
Oct 30 15:23:45.388581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388572    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy"
Oct 30 15:23:45.388859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388807    5064 cpu_manager.go:414] "containerMap: removing container" podUID="f55454ad-8f18-4afa-a3e5-b51944decbd2" containerName="maas-api"
Oct 30 15:23:45.388988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.388973    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55454ad-8f18-4afa-a3e5-b51944decbd2" containerName="maas-api"
Oct 30 15:23:45.389080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389070    5064 cpu_manager.go:414] "containerMap: removing container" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerName="main"
Oct 30 15:23:45.389153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389140    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerName="main"
Oct 30 15:23:45.389493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389475    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be869ab-5708-4180-80f7-801d1a44ec28" containerName="istio-proxy"
Oct 30 15:23:45.389619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389606    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55454ad-8f18-4afa-a3e5-b51944decbd2" containerName="maas-api"
Oct 30 15:23:45.389726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389716    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a93b058-4eed-4e3d-b397-192ea837fec6" containerName="main"
Oct 30 15:23:45.389801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389792    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="37762420-040d-446c-83b6-7d1e835bd5a3" containerName="main"
Oct 30 15:23:45.389900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389890    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced65a45-9933-453e-a2a1-846a3936d141" containerName="maas-api"
Oct 30 15:23:45.477331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.389984    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="942d2e88-2f2e-48b9-bf7c-11b0402667e2" containerName="istio-proxy"
Oct 30 15:23:45.485262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.484783    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.493314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.493277    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.494009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.493982    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.494220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.494202    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.494408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.494393    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.494554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.494540    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.494668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.494654    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppbl\" (UniqueName: \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.505809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.505762    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-859a8d\"/\"kube-root-ca.crt\""
Oct 30 15:23:45.507158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.506600    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-859a8d\"/\"test-model-859a8d-kserve-self-signed-certs\""
Oct 30 15:23:45.507316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.506632    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-859a8d\"/\"default-dockercfg-zhjxd\""
Oct 30 15:23:45.507416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.506912    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-859a8d\"/\"openshift-service-ca.crt\""
Oct 30 15:23:45.511971 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.510489    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:23:45.596468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596429    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.596657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596527    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.596657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596642    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.596745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596699    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.596797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596743    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppbl\" (UniqueName: \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.596950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596930    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.597026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.596986    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.597089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.597067    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.597183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.597162    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.599403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.599377    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.599981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.599952    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.622118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.622088    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppbl\" (UniqueName: \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\") pod \"test-model-859a8d-kserve-c9f685b8f-b5lqp\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") " pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:45.819262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:45.819173    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:46.142865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:46.142811    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:23:46.143228 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:23:46.143169    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fa7a30f_c95f_44df_8739_bdf46e7f3a23.slice/crio-6401c23882c7e48bbd33b4b87c25efe8c619fea1facfcb408fbe3053419b0fee WatchSource:0}: Error finding container 6401c23882c7e48bbd33b4b87c25efe8c619fea1facfcb408fbe3053419b0fee: Status 404 returned error can't find the container with id 6401c23882c7e48bbd33b4b87c25efe8c619fea1facfcb408fbe3053419b0fee
Oct 30 15:23:46.831681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:46.831637    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" event={"ID":"4fa7a30f-c95f-44df-8739-bdf46e7f3a23","Type":"ContainerStarted","Data":"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"}
Oct 30 15:23:46.831883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:46.831691    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" event={"ID":"4fa7a30f-c95f-44df-8739-bdf46e7f3a23","Type":"ContainerStarted","Data":"6401c23882c7e48bbd33b4b87c25efe8c619fea1facfcb408fbe3053419b0fee"}
Oct 30 15:23:46.876923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:46.876872    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" podStartSLOduration=1.8768544120000001 podStartE2EDuration="1.876854412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:23:46.876373163 +0000 UTC m=+2176.318263879" watchObservedRunningTime="2026-10-30 15:23:46.876854412 +0000 UTC m=+2176.318745126"
Oct 30 15:23:47.820721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:47.820681    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:48.622874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.622824    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:23:48.679367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679332    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679374    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679431    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679499    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679547    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679601    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679639    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:48.679883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679670    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.679883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679706    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:48.679883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679798    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.680098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679964    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:48.680098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.679981    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\") pod \"a508efa7-be16-48ac-9a84-d12f05a6ac29\" (UID: \"a508efa7-be16-48ac-9a84-d12f05a6ac29\") "
Oct 30 15:23:48.680202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.680115    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:48.680351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.680326    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:48.680852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.680810    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a508efa7-be16-48ac-9a84-d12f05a6ac29-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.680997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.680981    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.681122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.681100    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.681270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.681125    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.681270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.681141    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.682470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.682437    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:48.682558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.682473    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:23:48.682558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.682479    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:48.682558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.682507    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4" pod "a508efa7-be16-48ac-9a84-d12f05a6ac29" (UID: "a508efa7-be16-48ac-9a84-d12f05a6ac29"). InnerVolumeSpecName "kube-api-access-wfhf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:48.782560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.782511    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.782560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.782542    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.782560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.782556    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfhf4\" (UniqueName: \"kubernetes.io/projected/a508efa7-be16-48ac-9a84-d12f05a6ac29-kube-api-access-wfhf4\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.782560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.782569    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a508efa7-be16-48ac-9a84-d12f05a6ac29-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:48.849334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.849218    5064 generic.go:356] "Generic (PLEG): container finished" podID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerID="c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e" exitCode=0
Oct 30 15:23:48.849334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.849283    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"
Oct 30 15:23:48.849953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.849396    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" event={"ID":"a508efa7-be16-48ac-9a84-d12f05a6ac29","Type":"ContainerDied","Data":"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"}
Oct 30 15:23:48.849953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.849482    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82" event={"ID":"a508efa7-be16-48ac-9a84-d12f05a6ac29","Type":"ContainerDied","Data":"42e109202a8829362e46a0065e87ccee49485adb986f1adf1cafaf2bb93d26d3"}
Oct 30 15:23:48.849953 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.849516    5064 scope.go:122] "RemoveContainer" containerID="c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"
Oct 30 15:23:48.863040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.863017    5064 scope.go:122] "RemoveContainer" containerID="c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"
Oct 30 15:23:48.863378 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:48.863352    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e\": container with ID starting with c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e not found: ID does not exist" containerID="c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"
Oct 30 15:23:48.863470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.863389    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e"} err="failed to get container status \"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e\": rpc error: code = NotFound desc = could not find container \"c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e\": container with ID starting with c089549af579cf466cbcf97bc371f54f2b2f400cd24023cf1c07ffad643e585e not found: ID does not exist"
Oct 30 15:23:48.909437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.909402    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:23:48.913279 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:48.913255    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-jqz82"]
Oct 30 15:23:49.438648 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.438614    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:23:49.439411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.439395    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy"
Oct 30 15:23:49.439467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.439414    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy"
Oct 30 15:23:49.439536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.439527    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" containerName="istio-proxy"
Oct 30 15:23:49.444696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.444661    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.462595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.462105    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-13009a\"/\"kube-root-ca.crt\""
Oct 30 15:23:49.462595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.462261    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-13009a\"/\"default-dockercfg-h5p7w\""
Oct 30 15:23:49.462595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.462458    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-13009a\"/\"openshift-service-ca.crt\""
Oct 30 15:23:49.466769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.464645    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-13009a\"/\"test-model-13009a-kserve-self-signed-certs\""
Oct 30 15:23:49.466769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.465900    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:23:49.493255 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493216    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.493464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493338    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2h7j\" (UniqueName: \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.493556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493534    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.493622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493583    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.493687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493628    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.493687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.493664    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.595632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.595597    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.595803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.595719    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2h7j\" (UniqueName: \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.595872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.595854    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.595918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.595892    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.595972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.595922    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.596078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.596049    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.596353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.596330    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.596448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.596357    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.596448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.596391    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.598372 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.598352    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.598618 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.598596    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.621899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.621821    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2h7j\" (UniqueName: \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\") pod \"test-model-13009a-kserve-5d6ddbbc9f-6m4rc\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") " pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.782727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.782615    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:49.993563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.993522    5064 patch_prober.go:29] interesting pod/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:23:49.993977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:49.993597    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:23:50.182687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.182654    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:23:50.195208 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:23:50.195172    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd028e8f6_d6d7_4e07_980e_b2d587618020.slice/crio-db483c5a3e4c70e4cc1be24584b206d8b0c09a30ec9a26e4842450f7688b8692 WatchSource:0}: Error finding container db483c5a3e4c70e4cc1be24584b206d8b0c09a30ec9a26e4842450f7688b8692: Status 404 returned error can't find the container with id db483c5a3e4c70e4cc1be24584b206d8b0c09a30ec9a26e4842450f7688b8692
Oct 30 15:23:50.399131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.399093    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:23:50.584635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584602    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.584870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584689    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.584870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584786    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.584870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584812    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.584993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584875    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.584993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.584940    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.585090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585059    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.585271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585138    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:50.585271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585177    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:50.585271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585185    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.585271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585218    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:50.585271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585227    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:50.585714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585304    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\") pod \"c6991f3f-7592-4998-be53-204e55364fc3\" (UID: \"c6991f3f-7592-4998-be53-204e55364fc3\") "
Oct 30 15:23:50.585714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.585585    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:50.586178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.586150    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.586297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.586183    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6991f3f-7592-4998-be53-204e55364fc3-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.586297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.586198    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.586297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.586213    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.586297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.586227    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.587309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.587283    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:23:50.587394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.587320    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:50.587394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.587370    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "kube-api-access-6z6pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:50.587473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.587427    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo" pod "c6991f3f-7592-4998-be53-204e55364fc3" (UID: "c6991f3f-7592-4998-be53-204e55364fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:23:50.687728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.687634    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6pd\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-kube-api-access-6z6pd\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.687728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.687670    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6991f3f-7592-4998-be53-204e55364fc3-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.687728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.687681    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6991f3f-7592-4998-be53-204e55364fc3-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.687728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.687691    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6991f3f-7592-4998-be53-204e55364fc3-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:50.728381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.728325    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a508efa7-be16-48ac-9a84-d12f05a6ac29" path="/var/lib/kubelet/pods/a508efa7-be16-48ac-9a84-d12f05a6ac29/volumes"
Oct 30 15:23:50.896929 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.896885    5064 generic.go:356] "Generic (PLEG): container finished" podID="c6991f3f-7592-4998-be53-204e55364fc3" containerID="ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e" exitCode=0
Oct 30 15:23:50.897095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.896967    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"
Oct 30 15:23:50.897095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.896969    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerDied","Data":"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"}
Oct 30 15:23:50.897261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.897118    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn" event={"ID":"c6991f3f-7592-4998-be53-204e55364fc3","Type":"ContainerDied","Data":"4a5eaa5419e4ce0dd14f06a4a835419f8d77060cbd4bddcacfa63b497cdb5aae"}
Oct 30 15:23:50.897261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.897160    5064 scope.go:122] "RemoveContainer" containerID="ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"
Oct 30 15:23:50.899562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.899539    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" event={"ID":"d028e8f6-d6d7-4e07-980e-b2d587618020","Type":"ContainerStarted","Data":"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"}
Oct 30 15:23:50.899683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.899589    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" event={"ID":"d028e8f6-d6d7-4e07-980e-b2d587618020","Type":"ContainerStarted","Data":"db483c5a3e4c70e4cc1be24584b206d8b0c09a30ec9a26e4842450f7688b8692"}
Oct 30 15:23:50.910367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.910334    5064 scope.go:122] "RemoveContainer" containerID="462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"
Oct 30 15:23:50.922822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.922794    5064 scope.go:122] "RemoveContainer" containerID="ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"
Oct 30 15:23:50.923258 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:50.923227    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e\": container with ID starting with ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e not found: ID does not exist" containerID="ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"
Oct 30 15:23:50.923374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.923274    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e"} err="failed to get container status \"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e\": rpc error: code = NotFound desc = could not find container \"ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e\": container with ID starting with ca1b3e06f883744a694294b926514ae9e638132b7bde9b7114e59273cd6b562e not found: ID does not exist"
Oct 30 15:23:50.923374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.923303    5064 scope.go:122] "RemoveContainer" containerID="462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"
Oct 30 15:23:50.923685 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:50.923661    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3\": container with ID starting with 462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3 not found: ID does not exist" containerID="462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"
Oct 30 15:23:50.923759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.923693    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3"} err="failed to get container status \"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3\": rpc error: code = NotFound desc = could not find container \"462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3\": container with ID starting with 462fafa6b54bd0e96c71812e66987da6f0d66366538c38fc54f51d536e6acca3 not found: ID does not exist"
Oct 30 15:23:50.966205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:50.966097    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" podStartSLOduration=1.966079837 podStartE2EDuration="1.966079837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:23:50.962074135 +0000 UTC m=+2180.403964864" watchObservedRunningTime="2026-10-30 15:23:50.966079837 +0000 UTC m=+2180.407970554"
Oct 30 15:23:51.005287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:51.005251    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:23:51.018447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:51.018408    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-34218d-openshift-default-5d7c66f8c5-cpkpn"]
Oct 30 15:23:51.783482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:51.783447    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:52.731173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:52.731134    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6991f3f-7592-4998-be53-204e55364fc3" path="/var/lib/kubelet/pods/c6991f3f-7592-4998-be53-204e55364fc3/volumes"
Oct 30 15:23:54.190316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.190281    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:23:54.190817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.190531    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" podUID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" containerName="maas-api" containerID="cri-o://c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460" gracePeriod=30
Oct 30 15:23:54.746927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.746068    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:23:54.792512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.792466    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\" (UniqueName: \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\") pod \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") "
Oct 30 15:23:54.792730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.792685    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\") pod \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") "
Oct 30 15:23:54.792821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.792803    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\") pod \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") "
Oct 30 15:23:54.792911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.792869    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\") pod \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\" (UID: \"93be9abd-9452-42f3-ab8c-78e0882fb9b9\") "
Oct 30 15:23:54.818911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.795322    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca" pod "93be9abd-9452-42f3-ab8c-78e0882fb9b9" (UID: "93be9abd-9452-42f3-ab8c-78e0882fb9b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:23:54.818911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.815382    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs" pod "93be9abd-9452-42f3-ab8c-78e0882fb9b9" (UID: "93be9abd-9452-42f3-ab8c-78e0882fb9b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:54.824076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.823998    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls" pod "93be9abd-9452-42f3-ab8c-78e0882fb9b9" (UID: "93be9abd-9452-42f3-ab8c-78e0882fb9b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:23:54.824076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.824024    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx" pod "93be9abd-9452-42f3-ab8c-78e0882fb9b9" (UID: "93be9abd-9452-42f3-ab8c-78e0882fb9b9"). InnerVolumeSpecName "kube-api-access-dwphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:23:54.896126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.896086    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:54.896126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.896125    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93be9abd-9452-42f3-ab8c-78e0882fb9b9-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:54.896423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.896140    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93be9abd-9452-42f3-ab8c-78e0882fb9b9-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:54.896423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.896154    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwphx\" (UniqueName: \"kubernetes.io/projected/93be9abd-9452-42f3-ab8c-78e0882fb9b9-kube-api-access-dwphx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:23:54.942197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.942157    5064 generic.go:356] "Generic (PLEG): container finished" podID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" containerID="c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460" exitCode=0
Oct 30 15:23:54.942197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.942201    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"
Oct 30 15:23:54.942464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.942240    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" event={"ID":"93be9abd-9452-42f3-ab8c-78e0882fb9b9","Type":"ContainerDied","Data":"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"}
Oct 30 15:23:54.942464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.942309    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8" event={"ID":"93be9abd-9452-42f3-ab8c-78e0882fb9b9","Type":"ContainerDied","Data":"d626b3882a9dd4d00018e1d666aee24d1490e0fc2f31221512f662934b40ea33"}
Oct 30 15:23:54.942464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.942342    5064 scope.go:122] "RemoveContainer" containerID="c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"
Oct 30 15:23:54.953657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.953634    5064 scope.go:122] "RemoveContainer" containerID="c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"
Oct 30 15:23:54.954047 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:54.954027    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460\": container with ID starting with c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460 not found: ID does not exist" containerID="c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"
Oct 30 15:23:54.954123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:54.954061    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460"} err="failed to get container status \"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460\": rpc error: code = NotFound desc = could not find container \"c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460\": container with ID starting with c9dcde259ee3e88e0b66007e204b74022070f7559180dbc7d65fe75574515460 not found: ID does not exist"
Oct 30 15:23:55.003118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:55.003082    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:23:55.006599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:55.006575    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3bf5d2-799df859dc-94pn8"]
Oct 30 15:23:55.820380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:55.820340    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:55.833623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:55.833598    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:55.962468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:55.962435    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:23:56.724564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:56.724531    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" path="/var/lib/kubelet/pods/93be9abd-9452-42f3-ab8c-78e0882fb9b9/volumes"
Oct 30 15:23:59.632316 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:23:59.632276    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:23:59.784158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:59.784127    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:23:59.796946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:23:59.796903    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:24:00.031288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:00.031202    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:24:01.151453 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.151416    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:24:01.152222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152206    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:01.152275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152223    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:01.152275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152262    5064 cpu_manager.go:414] "containerMap: removing container" podUID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" containerName="maas-api"
Oct 30 15:24:01.152275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152267    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" containerName="maas-api"
Oct 30 15:24:01.152377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152278    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:01.152377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152287    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:01.152440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152412    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:01.152440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.152428    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="93be9abd-9452-42f3-ab8c-78e0882fb9b9" containerName="maas-api"
Oct 30 15:24:01.158227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.158200    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.177521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.177479    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:24:01.187208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.187182    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-af2072-openshift-default-dockercfg-w7x2b\""
Oct 30 15:24:01.214554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.214499    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.214751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.214581    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.214751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.214720    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69f89\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.214924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.214899    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.214994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.214970    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.215060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.215022    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.215143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.215129    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.215201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.215170    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.215309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.215287    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317066    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f89\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317256    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317316    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317377    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317420    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317473    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317608    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317651    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317735    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.317823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.317797    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.318088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.318063    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.318163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.318119    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.318426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.318405    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.319744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.319722    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.319920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.319902    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.343785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.343736    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69f89\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.343785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.343768    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.473461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.473374    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:01.801314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.801288    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:24:01.802310 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:01.802278    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7c6af9_cd63_4e58_87b4_f7fa57f10c39.slice/crio-c30dd6546aaae2bd6a4e18939ffdfec286faa816af7505288d264558b6cf65eb WatchSource:0}: Error finding container c30dd6546aaae2bd6a4e18939ffdfec286faa816af7505288d264558b6cf65eb: Status 404 returned error can't find the container with id c30dd6546aaae2bd6a4e18939ffdfec286faa816af7505288d264558b6cf65eb
Oct 30 15:24:01.805439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.805404    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:01.805548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.805490    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:01.805548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:01.805535    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:02.042915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.042878    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" event={"ID":"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39","Type":"ContainerStarted","Data":"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"}
Oct 30 15:24:02.043097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.042950    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" event={"ID":"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39","Type":"ContainerStarted","Data":"c30dd6546aaae2bd6a4e18939ffdfec286faa816af7505288d264558b6cf65eb"}
Oct 30 15:24:02.097022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.096915    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podStartSLOduration=1.096899103 podStartE2EDuration="1.096899103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:02.094382863 +0000 UTC m=+2191.536273581" watchObservedRunningTime="2026-10-30 15:24:02.096899103 +0000 UTC m=+2191.538789825"
Oct 30 15:24:02.301407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.301368    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:24:02.302266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.302225    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" podUID="6ce7edba-0871-467f-9da5-54bdcf295ed0" containerName="maas-api" containerID="cri-o://fb856cc1dd0f3c3b544cc0cf612a2a02486ba22356b7bf3b95b343a2adf5b0c8" gracePeriod=30
Oct 30 15:24:02.479845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.478286    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:02.480334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.480037    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:02.480334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:02.480088    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:03.266384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.266293    5064 generic.go:356] "Generic (PLEG): container finished" podID="6ce7edba-0871-467f-9da5-54bdcf295ed0" containerID="fb856cc1dd0f3c3b544cc0cf612a2a02486ba22356b7bf3b95b343a2adf5b0c8" exitCode=0
Oct 30 15:24:03.267108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.267061    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" event={"ID":"6ce7edba-0871-467f-9da5-54bdcf295ed0","Type":"ContainerDied","Data":"fb856cc1dd0f3c3b544cc0cf612a2a02486ba22356b7bf3b95b343a2adf5b0c8"}
Oct 30 15:24:03.267314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.267126    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr" event={"ID":"6ce7edba-0871-467f-9da5-54bdcf295ed0","Type":"ContainerDied","Data":"489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464"}
Oct 30 15:24:03.267314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.267155    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489931d3a62bfe38515b0ce3434b0d89baf2627c4a839f5b5512ba807991a464"
Oct 30 15:24:03.267500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.267480    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:24:03.460407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.460363    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\" (UniqueName: \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\") pod \"6ce7edba-0871-467f-9da5-54bdcf295ed0\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") "
Oct 30 15:24:03.461579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.461553    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") pod \"6ce7edba-0871-467f-9da5-54bdcf295ed0\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") "
Oct 30 15:24:03.461862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.461821    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\") pod \"6ce7edba-0871-467f-9da5-54bdcf295ed0\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") "
Oct 30 15:24:03.462630 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.462608    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\") pod \"6ce7edba-0871-467f-9da5-54bdcf295ed0\" (UID: \"6ce7edba-0871-467f-9da5-54bdcf295ed0\") "
Oct 30 15:24:03.463082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.463052    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca" pod "6ce7edba-0871-467f-9da5-54bdcf295ed0" (UID: "6ce7edba-0871-467f-9da5-54bdcf295ed0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:24:03.463705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.463682    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce7edba-0871-467f-9da5-54bdcf295ed0-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:03.464573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.464544    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs" pod "6ce7edba-0871-467f-9da5-54bdcf295ed0" (UID: "6ce7edba-0871-467f-9da5-54bdcf295ed0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:24:03.464678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.464600    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l" pod "6ce7edba-0871-467f-9da5-54bdcf295ed0" (UID: "6ce7edba-0871-467f-9da5-54bdcf295ed0"). InnerVolumeSpecName "kube-api-access-lkf4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:03.465153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.465130    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls" pod "6ce7edba-0871-467f-9da5-54bdcf295ed0" (UID: "6ce7edba-0871-467f-9da5-54bdcf295ed0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:24:03.475028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.474996    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:03.475139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.475054    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:03.565454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.565367    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkf4l\" (UniqueName: \"kubernetes.io/projected/6ce7edba-0871-467f-9da5-54bdcf295ed0-kube-api-access-lkf4l\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:03.565454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.565396    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:03.565454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:03.565407    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce7edba-0871-467f-9da5-54bdcf295ed0-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:04.274906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.274863    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"
Oct 30 15:24:04.344805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.344773    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:24:04.359867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.359816    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765228-74875597c8-ksnfr"]
Oct 30 15:24:04.474170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.474130    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:04.474608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.474189    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:04.745948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:04.745911    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce7edba-0871-467f-9da5-54bdcf295ed0" path="/var/lib/kubelet/pods/6ce7edba-0871-467f-9da5-54bdcf295ed0/volumes"
Oct 30 15:24:05.473927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:05.473880    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:05.474122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:05.473958    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:06.473760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:06.473726    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:06.474180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:06.473789    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:07.474448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:07.474409    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:07.475004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:07.474480    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:08.474686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:08.474641    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 15:24:08.475197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:08.474714    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 15:24:09.479513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:09.479480    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:09.479979 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:09.479963    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:09.480703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:09.480681    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:24:10.774184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:10.774083    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:24:10.774672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:10.774349    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" containerID="cri-o://2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc" gracePeriod=30
Oct 30 15:24:12.680072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.680037    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:24:12.680806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.680789    5064 cpu_manager.go:414] "containerMap: removing container" podUID="6ce7edba-0871-467f-9da5-54bdcf295ed0" containerName="maas-api"
Oct 30 15:24:12.680806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.680804    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce7edba-0871-467f-9da5-54bdcf295ed0" containerName="maas-api"
Oct 30 15:24:12.680983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.680971    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce7edba-0871-467f-9da5-54bdcf295ed0" containerName="maas-api"
Oct 30 15:24:12.681048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.680990    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6991f3f-7592-4998-be53-204e55364fc3" containerName="istio-proxy"
Oct 30 15:24:12.686216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.686193    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.706145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.706114    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:24:12.714446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.714415    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-985bf6\"/\"default-dockercfg-zzsrf\""
Oct 30 15:24:12.714446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.714415    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-985bf6\"/\"rate-test-model-985bf6-kserve-self-signed-certs\""
Oct 30 15:24:12.718207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.718184    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-985bf6\"/\"openshift-service-ca.crt\""
Oct 30 15:24:12.718351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.718236    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-985bf6\"/\"kube-root-ca.crt\""
Oct 30 15:24:12.773481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.773442    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.773683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.773521    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lf2\" (UniqueName: \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.773683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.773613    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.773822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.773773    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.773898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.773872    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.774030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.774005    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875506    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875596    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lf2\" (UniqueName: \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875664    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875770    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875797    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.875908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.875890    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.876173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.876147    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.876309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.876196    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.876309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.876233    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.878072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.878051    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.878494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.878473    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:12.903706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:12.903672    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lf2\" (UniqueName: \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\") pod \"rate-test-model-985bf6-kserve-7b79dc7685-q5gg2\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") " pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:13.008558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:13.008175    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:13.415698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:13.415669    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:24:13.416287 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:13.416256    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd27a4ba_e27d_4bf2_ab28_018cc256c4f3.slice/crio-caa191e0508ca14b2344656ce6cacd5726517e1e4da7291be5400fe037d74654 WatchSource:0}: Error finding container caa191e0508ca14b2344656ce6cacd5726517e1e4da7291be5400fe037d74654: Status 404 returned error can't find the container with id caa191e0508ca14b2344656ce6cacd5726517e1e4da7291be5400fe037d74654
Oct 30 15:24:14.362041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:14.361997    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" event={"ID":"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3","Type":"ContainerStarted","Data":"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"}
Oct 30 15:24:14.362473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:14.362057    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" event={"ID":"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3","Type":"ContainerStarted","Data":"caa191e0508ca14b2344656ce6cacd5726517e1e4da7291be5400fe037d74654"}
Oct 30 15:24:14.411806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:14.411748    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" podStartSLOduration=2.411729355 podStartE2EDuration="2.411729355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:14.408011388 +0000 UTC m=+2203.849902108" watchObservedRunningTime="2026-10-30 15:24:14.411729355 +0000 UTC m=+2203.853620072"
Oct 30 15:24:15.010014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:15.009939    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:15.855186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:15.855153    5064 patch_prober.go:29] interesting pod/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 15:24:15.855594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:15.855211    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 15:24:16.072625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.072598    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:24:16.140561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.140517    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.140802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.140684    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.140802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.140727    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.141012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.140885    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.141012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.140980    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.141129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.141081    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.141188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.141164    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.141242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.141228    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.142068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.141343    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\") pod \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\" (UID: \"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d\") "
Oct 30 15:24:16.142423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.142361    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:16.142591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.142571    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:16.143476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.143447    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:16.143604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.143478    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:16.143674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.143624    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:24:16.143752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.143685    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:16.145965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.145928    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:24:16.145965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.145940    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:16.146112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.146063    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg" pod "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" (UID: "a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d"). InnerVolumeSpecName "kube-api-access-pjsjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:16.244081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244048    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244077    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244086    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244096    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjsjg\" (UniqueName: \"kubernetes.io/projected/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-kube-api-access-pjsjg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244106    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244116    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244124    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244132    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.244330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.244141    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:16.379616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.379582    5064 generic.go:356] "Generic (PLEG): container finished" podID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerID="2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc" exitCode=0
Oct 30 15:24:16.379807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.379645    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"
Oct 30 15:24:16.379807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.379662    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerDied","Data":"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"}
Oct 30 15:24:16.379807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.379715    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt" event={"ID":"a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d","Type":"ContainerDied","Data":"2f0289245aa73c350de6d9fb183e18029577ac8bb0d40425612fb79bc3f8f437"}
Oct 30 15:24:16.379807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.379737    5064 scope.go:122] "RemoveContainer" containerID="2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"
Oct 30 15:24:16.392683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.392665    5064 scope.go:122] "RemoveContainer" containerID="d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"
Oct 30 15:24:16.403547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.403522    5064 scope.go:122] "RemoveContainer" containerID="2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"
Oct 30 15:24:16.403895 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:24:16.403872    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc\": container with ID starting with 2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc not found: ID does not exist" containerID="2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"
Oct 30 15:24:16.404000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.403911    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc"} err="failed to get container status \"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc\": rpc error: code = NotFound desc = could not find container \"2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc\": container with ID starting with 2372b31221eae8fb5af7f34f229d61376d2e1e6422c2744b963c7849a5a442bc not found: ID does not exist"
Oct 30 15:24:16.404000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.403936    5064 scope.go:122] "RemoveContainer" containerID="d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"
Oct 30 15:24:16.404224 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:24:16.404199    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5\": container with ID starting with d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5 not found: ID does not exist" containerID="d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"
Oct 30 15:24:16.404266 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.404231    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5"} err="failed to get container status \"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5\": rpc error: code = NotFound desc = could not find container \"d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5\": container with ID starting with d9a3fac62f6ae72ae4913f6bf6f36365f209a47c30a51b6619aaf82a683189d5 not found: ID does not exist"
Oct 30 15:24:16.439379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.439343    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:24:16.453697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.453657    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-765228-openshift-default-86f7dbd8b9-h76zt"]
Oct 30 15:24:16.726182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:16.726092    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" path="/var/lib/kubelet/pods/a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d/volumes"
Oct 30 15:24:23.009186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:23.009152    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:23.022972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:23.022946    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:23.465038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:23.465006    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:24:29.428941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:29.428891    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:24:29.429530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:29.429124    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" podUID="5130c9cf-658b-44d1-92f9-53f8a21924a4" containerName="maas-api" containerID="cri-o://6d246255201e2ea19e025d8897fdff65cf6d699b73a49e038401a2fa6fd7dfe1" gracePeriod=30
Oct 30 15:24:29.572936 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:29.572883    5064 generic.go:356] "Generic (PLEG): container finished" podID="5130c9cf-658b-44d1-92f9-53f8a21924a4" containerID="6d246255201e2ea19e025d8897fdff65cf6d699b73a49e038401a2fa6fd7dfe1" exitCode=0
Oct 30 15:24:29.573079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:29.572949    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" event={"ID":"5130c9cf-658b-44d1-92f9-53f8a21924a4","Type":"ContainerDied","Data":"6d246255201e2ea19e025d8897fdff65cf6d699b73a49e038401a2fa6fd7dfe1"}
Oct 30 15:24:29.647121 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:24:29.647082    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:24:30.087865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.086966    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:24:30.163312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.163228    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\") pod \"5130c9cf-658b-44d1-92f9-53f8a21924a4\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") "
Oct 30 15:24:30.163589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.163570    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\") pod \"5130c9cf-658b-44d1-92f9-53f8a21924a4\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") "
Oct 30 15:24:30.163725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.163709    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\" (UniqueName: \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\") pod \"5130c9cf-658b-44d1-92f9-53f8a21924a4\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") "
Oct 30 15:24:30.164094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.164075    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\") pod \"5130c9cf-658b-44d1-92f9-53f8a21924a4\" (UID: \"5130c9cf-658b-44d1-92f9-53f8a21924a4\") "
Oct 30 15:24:30.166463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.166431    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca" pod "5130c9cf-658b-44d1-92f9-53f8a21924a4" (UID: "5130c9cf-658b-44d1-92f9-53f8a21924a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:24:30.176499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.175144    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg" pod "5130c9cf-658b-44d1-92f9-53f8a21924a4" (UID: "5130c9cf-658b-44d1-92f9-53f8a21924a4"). InnerVolumeSpecName "kube-api-access-swjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:30.176499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.175203    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls" pod "5130c9cf-658b-44d1-92f9-53f8a21924a4" (UID: "5130c9cf-658b-44d1-92f9-53f8a21924a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:24:30.176499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.175341    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs" pod "5130c9cf-658b-44d1-92f9-53f8a21924a4" (UID: "5130c9cf-658b-44d1-92f9-53f8a21924a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:24:30.267092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.267054    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:30.267092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.267093    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5130c9cf-658b-44d1-92f9-53f8a21924a4-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:30.267297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.267107    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5130c9cf-658b-44d1-92f9-53f8a21924a4-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:30.267297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.267120    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjbg\" (UniqueName: \"kubernetes.io/projected/5130c9cf-658b-44d1-92f9-53f8a21924a4-kube-api-access-swjbg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:30.583756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.583659    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"
Oct 30 15:24:30.583756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.583685    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2" event={"ID":"5130c9cf-658b-44d1-92f9-53f8a21924a4","Type":"ContainerDied","Data":"f910f16603f1294658437796dd6454069b42826f96373f8ef3ae24a32e005888"}
Oct 30 15:24:30.584426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.583780    5064 scope.go:122] "RemoveContainer" containerID="6d246255201e2ea19e025d8897fdff65cf6d699b73a49e038401a2fa6fd7dfe1"
Oct 30 15:24:30.644969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.644919    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:24:30.649138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.649105    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d84bc6-68d79c576f-xrmj2"]
Oct 30 15:24:30.727346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:30.727314    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5130c9cf-658b-44d1-92f9-53f8a21924a4" path="/var/lib/kubelet/pods/5130c9cf-658b-44d1-92f9-53f8a21924a4/volumes"
Oct 30 15:24:32.410280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:32.410245    5064 scope.go:122] "RemoveContainer" containerID="fb856cc1dd0f3c3b544cc0cf612a2a02486ba22356b7bf3b95b343a2adf5b0c8"
Oct 30 15:24:37.434769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:37.434727    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:24:37.435284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:37.434973    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" containerID="cri-o://7400023a7dd4894a5857a67bffe0c600d357702ef75fe92012256ed7ee563698" gracePeriod=30
Oct 30 15:24:38.674215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.674173    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:24:38.675938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.675909    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676101    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676146    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676156    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676190    5064 cpu_manager.go:414] "containerMap: removing container" podUID="5130c9cf-658b-44d1-92f9-53f8a21924a4" containerName="maas-api"
Oct 30 15:24:38.676224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676198    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5130c9cf-658b-44d1-92f9-53f8a21924a4" containerName="maas-api"
Oct 30 15:24:38.676468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676452    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676474    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13a4c3c-07b3-4ae9-94cc-965a41fdeb4d" containerName="istio-proxy"
Oct 30 15:24:38.676504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.676486    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="5130c9cf-658b-44d1-92f9-53f8a21924a4" containerName="maas-api"
Oct 30 15:24:38.683439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.683398    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.698029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.697430    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:24:38.705267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.705024    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bcd8d6-openshift-default-dockercfg-vhgnw\""
Oct 30 15:24:38.739956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.739916    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740037    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740133    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740211    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740238    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740256    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740326    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740378    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.740440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.740423    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fwf\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841388 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841354    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fwf\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841482    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841547    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841671    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841753 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841739    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841776    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841805    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.841888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841857    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841900    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.841997    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.842032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.842235    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.842346    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.842665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.842643    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.844251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.844229    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.844519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.844501    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.940931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.923475    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:38.940931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:38.924123    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fwf\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:39.010867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.010122    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:39.443768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.443738    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:24:39.445206 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:39.445172    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec42a1b4_40bd_4f6f_8ac7_13be20aab0f5.slice/crio-00a8819f36bf48d217392799ff4743fe2bc5006eb1a4586ff361d4674c91a9d1 WatchSource:0}: Error finding container 00a8819f36bf48d217392799ff4743fe2bc5006eb1a4586ff361d4674c91a9d1: Status 404 returned error can't find the container with id 00a8819f36bf48d217392799ff4743fe2bc5006eb1a4586ff361d4674c91a9d1
Oct 30 15:24:39.448108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.448060    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:39.448201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.448142    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:39.448201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.448179    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:39.671114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.671078    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerStarted","Data":"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"}
Oct 30 15:24:39.671324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.671137    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerStarted","Data":"00a8819f36bf48d217392799ff4743fe2bc5006eb1a4586ff361d4674c91a9d1"}
Oct 30 15:24:39.725890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:39.725750    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podStartSLOduration=1.725729469 podStartE2EDuration="1.725729469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:39.721067414 +0000 UTC m=+2229.162958157" watchObservedRunningTime="2026-10-30 15:24:39.725729469 +0000 UTC m=+2229.167620187"
Oct 30 15:24:40.010740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.010642    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:40.017192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.017169    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:40.589420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.589372    5064 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:24:40.589580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.589442    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:24:40.678997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.678972    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:40.679857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:40.679798    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:24:42.698033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.697927    5064 generic.go:356] "Generic (PLEG): container finished" podID="62f3acbd-0587-4145-805f-a0b142364dd4" containerID="7400023a7dd4894a5857a67bffe0c600d357702ef75fe92012256ed7ee563698" exitCode=0
Oct 30 15:24:42.698033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.698015    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" event={"ID":"62f3acbd-0587-4145-805f-a0b142364dd4","Type":"ContainerDied","Data":"7400023a7dd4894a5857a67bffe0c600d357702ef75fe92012256ed7ee563698"}
Oct 30 15:24:42.698545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.698071    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t" event={"ID":"62f3acbd-0587-4145-805f-a0b142364dd4","Type":"ContainerDied","Data":"1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c"}
Oct 30 15:24:42.698545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.698093    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1798529652c91c6024f87221248b3e74adc1e61c72e8d7d0634cd84e0ec4d03c"
Oct 30 15:24:42.718934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.718904    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:24:42.800505 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800472    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800519    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800560    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800693 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800686    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800728    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800757    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800783    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.800892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800799    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:42.800892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800820    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.801159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.800956    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\") pod \"62f3acbd-0587-4145-805f-a0b142364dd4\" (UID: \"62f3acbd-0587-4145-805f-a0b142364dd4\") "
Oct 30 15:24:42.801159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801112    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:42.801159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801122    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:42.801382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801362    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:42.801534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801504    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:24:42.801919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801888    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.801919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801913    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.802107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801927    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f3acbd-0587-4145-805f-a0b142364dd4-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.802107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801941    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.802107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.801954    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.803095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.803066    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:24:42.803540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.803510    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:24:42.803623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.803547    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "kube-api-access-hlndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:42.803623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.803572    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token" pod "62f3acbd-0587-4145-805f-a0b142364dd4" (UID: "62f3acbd-0587-4145-805f-a0b142364dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:24:42.903889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.903806    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f3acbd-0587-4145-805f-a0b142364dd4-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.903889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.903888    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.903889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.903900    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f3acbd-0587-4145-805f-a0b142364dd4-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:42.904151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:42.903909    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlndc\" (UniqueName: \"kubernetes.io/projected/62f3acbd-0587-4145-805f-a0b142364dd4-kube-api-access-hlndc\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:24:43.706946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:43.706914    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"
Oct 30 15:24:43.778692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:43.778654    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:24:43.792219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:43.792182    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-fss9t"]
Oct 30 15:24:44.127552 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.127517    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:24:44.128250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.128235    5064 cpu_manager.go:414] "containerMap: removing container" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy"
Oct 30 15:24:44.128250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.128250    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy"
Oct 30 15:24:44.128394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.128384    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" containerName="istio-proxy"
Oct 30 15:24:44.133346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.133319    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.150416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.150383    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:24:44.151462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.151435    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bcd8d6-dockercfg-xxqkx\""
Oct 30 15:24:44.151579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.151534    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bcd8d6\""
Oct 30 15:24:44.312964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.312902    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:24:44.395315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.395223    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:24:44.395733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.395710    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.437864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.430777    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.437864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.436302    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hgs\" (UniqueName: \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.538332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.538298    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.538556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.538376    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hgs\" (UniqueName: \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.538556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.538509    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddw9\" (UniqueName: \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.538692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.538600    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.539551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.539529    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.563698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.563661    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hgs\" (UniqueName: \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\") pod \"payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.640510 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.640470    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.640707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.640581    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddw9\" (UniqueName: \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.641377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.641349    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.666315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.666241    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddw9\" (UniqueName: \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\") pod \"payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:44.724497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.724463    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f3acbd-0587-4145-805f-a0b142364dd4" path="/var/lib/kubelet/pods/62f3acbd-0587-4145-805f-a0b142364dd4/volumes"
Oct 30 15:24:44.748930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.748896    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:44.789158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:44.789120    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:45.066702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.066667    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:24:45.067599 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:45.067561    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e99d3d_66dc_42bd_96db_8b5ae46c592f.slice/crio-26a05eb4c990948900745553f6628660d3db18cba679e4e5308784af2cc9755c WatchSource:0}: Error finding container 26a05eb4c990948900745553f6628660d3db18cba679e4e5308784af2cc9755c: Status 404 returned error can't find the container with id 26a05eb4c990948900745553f6628660d3db18cba679e4e5308784af2cc9755c
Oct 30 15:24:45.123722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.123697    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:24:45.131476 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:45.131443    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0027ba_5a92_4c25_940a_ce550abcfa9f.slice/crio-3bfffbf804ae940c24589f0c159f2aca8c7d9d690a5ff2993a1cbdb66fe26575 WatchSource:0}: Error finding container 3bfffbf804ae940c24589f0c159f2aca8c7d9d690a5ff2993a1cbdb66fe26575: Status 404 returned error can't find the container with id 3bfffbf804ae940c24589f0c159f2aca8c7d9d690a5ff2993a1cbdb66fe26575
Oct 30 15:24:45.724020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.723974    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" event={"ID":"2b0027ba-5a92-4c25-940a-ce550abcfa9f","Type":"ContainerStarted","Data":"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"}
Oct 30 15:24:45.724333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.724046    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" event={"ID":"2b0027ba-5a92-4c25-940a-ce550abcfa9f","Type":"ContainerStarted","Data":"3bfffbf804ae940c24589f0c159f2aca8c7d9d690a5ff2993a1cbdb66fe26575"}
Oct 30 15:24:45.724333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.724320    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:45.726054 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.726026    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" event={"ID":"50e99d3d-66dc-42bd-96db-8b5ae46c592f","Type":"ContainerStarted","Data":"ef7741ae60d5e350f001ebcffc4cbd538fbb583753bb421821b8f17648c53443"}
Oct 30 15:24:45.726408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.726072    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" event={"ID":"50e99d3d-66dc-42bd-96db-8b5ae46c592f","Type":"ContainerStarted","Data":"26a05eb4c990948900745553f6628660d3db18cba679e4e5308784af2cc9755c"}
Oct 30 15:24:45.726408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.726093    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:45.772912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.772854    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" podStartSLOduration=1.77282386 podStartE2EDuration="1.77282386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:45.769548487 +0000 UTC m=+2235.211439205" watchObservedRunningTime="2026-10-30 15:24:45.77282386 +0000 UTC m=+2235.214714578"
Oct 30 15:24:45.813839 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:45.813767    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" podStartSLOduration=1.813749959 podStartE2EDuration="1.813749959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:45.811750284 +0000 UTC m=+2235.253641002" watchObservedRunningTime="2026-10-30 15:24:45.813749959 +0000 UTC m=+2235.255640676"
Oct 30 15:24:46.900607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:46.900576    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:24:46.905390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:46.905362    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:46.938215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:46.938181    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c5235-openshift-default-dockercfg-r8knf\""
Oct 30 15:24:46.942609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:46.942572    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:24:47.080625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080579    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hzl\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.080824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080706    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.080824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080786    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.080962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080909    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.080962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080938    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.081048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.080972    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.081149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.081123    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.081241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.081226    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.081291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.081251    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182329    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182383    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182464    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182491    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182542    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182634    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182716    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182743    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182890    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hzl\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.182985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.182969    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.183091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.183014    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.183091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.183060    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.183271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.183251    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.183697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.183674    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.184956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.184933    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.185371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.185349    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.207623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.207591    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.212622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.212526    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hzl\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.221529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.221491    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:47.551005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.550973    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:24:47.553436 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:47.553400    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecca84f_ccda_4c77_978f_b7568ad38457.slice/crio-67c88c6c723ba96b0a8cd6f64fd99eb247e9f9f59fe95d758600faac15ba5652 WatchSource:0}: Error finding container 67c88c6c723ba96b0a8cd6f64fd99eb247e9f9f59fe95d758600faac15ba5652: Status 404 returned error can't find the container with id 67c88c6c723ba96b0a8cd6f64fd99eb247e9f9f59fe95d758600faac15ba5652
Oct 30 15:24:47.556069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.556032    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:47.556169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.556116    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:47.556169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.556143    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:24:47.743741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.743702    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" event={"ID":"decca84f-ccda-4c77-978f-b7568ad38457","Type":"ContainerStarted","Data":"261c42fdd32737731bc46bbb912aafc756cc89f6deb5435d52bc7561415c66a6"}
Oct 30 15:24:47.743982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.743762    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" event={"ID":"decca84f-ccda-4c77-978f-b7568ad38457","Type":"ContainerStarted","Data":"67c88c6c723ba96b0a8cd6f64fd99eb247e9f9f59fe95d758600faac15ba5652"}
Oct 30 15:24:47.813907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:47.791610    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podStartSLOduration=1.791590153 podStartE2EDuration="1.791590153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:47.791045308 +0000 UTC m=+2237.232936051" watchObservedRunningTime="2026-10-30 15:24:47.791590153 +0000 UTC m=+2237.233480875"
Oct 30 15:24:48.236310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:48.236269    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:48.240288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:48.240251    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:48.240469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:48.240315    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:49.230856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.224368    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:49.230856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.224447    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:49.560146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.560061    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:24:49.564999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.564972    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.584191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.584153    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:24:49.592040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.592001    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-609709\"/\"kube-root-ca.crt\""
Oct 30 15:24:49.592212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.592067    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-609709\"/\"openshift-service-ca.crt\""
Oct 30 15:24:49.595938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.595908    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-609709\"/\"default-dockercfg-h7c4l\""
Oct 30 15:24:49.596087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.596027    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-609709\"/\"rate-test-model-609709-kserve-self-signed-certs\""
Oct 30 15:24:49.664670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.664636    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.664879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.664755    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g52n\" (UniqueName: \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.664879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.664816    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.664997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.664973    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.665063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.665050    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.665096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.665072    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.766229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.766195    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.766673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.766649    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.766822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.766766    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.766822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.766796    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.767073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.767029    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.767168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.767121    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.767229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.767185    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.767229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.767207    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g52n\" (UniqueName: \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.767442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.767419    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.769353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.769331    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.769452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.769433    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.819809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.819693    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g52n\" (UniqueName: \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\") pod \"rate-test-model-609709-kserve-5c79f98d45-98wb7\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") " pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:49.885449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:49.885125    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:50.239343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.239300    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:50.239533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.239372    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:50.299653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.299625    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:24:50.300449 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:24:50.300419    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3baf93c8_b179_4a6f_a84b_19f3257fa812.slice/crio-36ef77a8ed936df567abdd26e2f25f119b4fc49a0adb53e12ce76058a7d8a971 WatchSource:0}: Error finding container 36ef77a8ed936df567abdd26e2f25f119b4fc49a0adb53e12ce76058a7d8a971: Status 404 returned error can't find the container with id 36ef77a8ed936df567abdd26e2f25f119b4fc49a0adb53e12ce76058a7d8a971
Oct 30 15:24:50.770967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.770923    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" event={"ID":"3baf93c8-b179-4a6f-a84b-19f3257fa812","Type":"ContainerStarted","Data":"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"}
Oct 30 15:24:50.770967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.770970    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" event={"ID":"3baf93c8-b179-4a6f-a84b-19f3257fa812","Type":"ContainerStarted","Data":"36ef77a8ed936df567abdd26e2f25f119b4fc49a0adb53e12ce76058a7d8a971"}
Oct 30 15:24:50.841896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.841805    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" podStartSLOduration=1.841785241 podStartE2EDuration="1.841785241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:50.8212423 +0000 UTC m=+2240.263133018" watchObservedRunningTime="2026-10-30 15:24:50.841785241 +0000 UTC m=+2240.283675958"
Oct 30 15:24:50.886563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:50.886536    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:51.224912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:51.224229    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:51.224912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:51.224305    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:52.234886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:52.227297    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:52.234886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:52.227366    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:53.238409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:53.238365    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 15:24:53.238871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:53.238441    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 15:24:54.417742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:54.417700    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:54.418220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:54.418167    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:54.419124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:54.419101    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:24:56.736343 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:56.736316    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:24:56.736755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:56.736378    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:24:59.009481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:59.009442    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:24:59.010261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:59.010230    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" containerID="cri-o://63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf" gracePeriod=30
Oct 30 15:24:59.698952 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:24:59.698902    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:24:59.886470 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:59.886426    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:24:59.900011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:24:59.899980    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:25:00.257599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.257555    5064 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:25:00.258073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.257633    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:25:00.299786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.299749    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:25:00.309730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.309702    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.326748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.326719    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3f20f9-openshift-default-dockercfg-g9ft8\""
Oct 30 15:25:00.333821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.333792    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:25:00.442115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.442066    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.442426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.442403    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.442592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.442576    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.442757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.442740    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.442932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.442913    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.443127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.443111    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlh4l\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.450863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.443309    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.451267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.451234    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.451570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.451539    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566057    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566131    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566236    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlh4l\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566330    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566463    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566564    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566662    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566747    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.566845    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.567336    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.568195    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.568870    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.572925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.569245    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.573729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.573524    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.607861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.604196    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.632688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.622445    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.632688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.623024    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.632688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.625293    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlh4l\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.642007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.640780    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:00.864193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:00.864102    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:25:01.047812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.047780    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:25:01.048535 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:25:01.048508    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd275e859_a3c6_4d17_97b9_3bd90f0aa285.slice/crio-aa04161a28e64970e9807b5486884fa53468b69c34de449f5f8cb30644235858 WatchSource:0}: Error finding container aa04161a28e64970e9807b5486884fa53468b69c34de449f5f8cb30644235858: Status 404 returned error can't find the container with id aa04161a28e64970e9807b5486884fa53468b69c34de449f5f8cb30644235858
Oct 30 15:25:01.051539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.051503    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:01.051607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.051579    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:01.051645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.051609    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:01.860783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.860737    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerStarted","Data":"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"}
Oct 30 15:25:01.861369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.860805    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerStarted","Data":"aa04161a28e64970e9807b5486884fa53468b69c34de449f5f8cb30644235858"}
Oct 30 15:25:01.910504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:01.910445    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podStartSLOduration=1.910430823 podStartE2EDuration="1.910430823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:01.907573223 +0000 UTC m=+2251.349463940" watchObservedRunningTime="2026-10-30 15:25:01.910430823 +0000 UTC m=+2251.352321539"
Oct 30 15:25:02.642011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:02.641973    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:02.649021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:02.648985    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:02.869411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:02.869368    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:02.870429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:02.870403    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:25:04.310937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.310895    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:25:04.444353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.444315    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.444574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.444394    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.444574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.444441    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.445927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.444523    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.446186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.445460    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:04.446293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446092    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:04.446371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446179    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.446674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446644    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.446818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446743    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.446818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446787    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.447007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446850    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\") pod \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\" (UID: \"d4f2b11c-2539-436a-8264-c0db6dfb87a0\") "
Oct 30 15:25:04.447638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.447614    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.447638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.447639    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.448237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.446460    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:04.448341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.448208    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:25:04.448733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.448704    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:04.450508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.450470    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "kube-api-access-5nbnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:25:04.450767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.450742    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:25:04.450942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.450916    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:25:04.452206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.452176    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy" pod "d4f2b11c-2539-436a-8264-c0db6dfb87a0" (UID: "d4f2b11c-2539-436a-8264-c0db6dfb87a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549820    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549914    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549931    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549946    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nbnh\" (UniqueName: \"kubernetes.io/projected/d4f2b11c-2539-436a-8264-c0db6dfb87a0-kube-api-access-5nbnh\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549960    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549974    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.550925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.549986    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b11c-2539-436a-8264-c0db6dfb87a0-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:04.885464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.885431    5064 generic.go:356] "Generic (PLEG): container finished" podID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerID="63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf" exitCode=0
Oct 30 15:25:04.885680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.885492    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"
Oct 30 15:25:04.885680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.885521    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" event={"ID":"d4f2b11c-2539-436a-8264-c0db6dfb87a0","Type":"ContainerDied","Data":"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"}
Oct 30 15:25:04.885680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.885581    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7" event={"ID":"d4f2b11c-2539-436a-8264-c0db6dfb87a0","Type":"ContainerDied","Data":"8e4ec2c9b4c330871c704d4fbdbd4671906ec9b30a041bf13cc5ccaa59879a39"}
Oct 30 15:25:04.885680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.885609    5064 scope.go:122] "RemoveContainer" containerID="63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"
Oct 30 15:25:04.897089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.897067    5064 scope.go:122] "RemoveContainer" containerID="63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"
Oct 30 15:25:04.897423 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:25:04.897393    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf\": container with ID starting with 63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf not found: ID does not exist" containerID="63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"
Oct 30 15:25:04.897500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.897430    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf"} err="failed to get container status \"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf\": rpc error: code = NotFound desc = could not find container \"63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf\": container with ID starting with 63dee722e52bf3c5f6304d01726f4a375777125fb68a134b830d217e0e190edf not found: ID does not exist"
Oct 30 15:25:04.960111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.960076    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:25:04.972843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:04.972801    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-4zqh7"]
Oct 30 15:25:06.725006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:06.724975    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" path="/var/lib/kubelet/pods/d4f2b11c-2539-436a-8264-c0db6dfb87a0/volumes"
Oct 30 15:25:24.333234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.333197    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:25:24.334012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.333995    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy"
Oct 30 15:25:24.334072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.334013    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy"
Oct 30 15:25:24.334138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.334127    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f2b11c-2539-436a-8264-c0db6dfb87a0" containerName="istio-proxy"
Oct 30 15:25:24.339625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.339603    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.359163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.359128    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:25:24.371114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.371082    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-bcd8d6-serving-cert\""
Oct 30 15:25:24.468825 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.468787    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.469106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.469078    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.469221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.469197    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mw8\" (UniqueName: \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.469361 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.469342    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.573020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.571186    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.573020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.571311    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.573020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.571507    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.573020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.571601    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mw8\" (UniqueName: \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.574494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.574433    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.625076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.624274    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mw8\" (UniqueName: \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.625076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.625011    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.625456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.625424    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.661481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.661450    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:24.983803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:24.983766    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:25:24.985427 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:25:24.985402    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b560ba_e043_4424_9fc4_962f42f0d321.slice/crio-b2d5e871baaf9af529c4d54053a4e96c79dd6fb9afcccdd0a0bcb5f995ead7da WatchSource:0}: Error finding container b2d5e871baaf9af529c4d54053a4e96c79dd6fb9afcccdd0a0bcb5f995ead7da: Status 404 returned error can't find the container with id b2d5e871baaf9af529c4d54053a4e96c79dd6fb9afcccdd0a0bcb5f995ead7da
Oct 30 15:25:25.254859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:25.254753    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" event={"ID":"f9b560ba-e043-4424-9fc4-962f42f0d321","Type":"ContainerStarted","Data":"b2d5e871baaf9af529c4d54053a4e96c79dd6fb9afcccdd0a0bcb5f995ead7da"}
Oct 30 15:25:25.397549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:25.397466    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 15:25:25.398130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:25.397885    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" containerID="cri-o://37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c" gracePeriod=30
Oct 30 15:25:26.263413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.263380    5064 generic.go:356] "Generic (PLEG): container finished" podID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerID="66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1" exitCode=0
Oct 30 15:25:26.263639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.263482    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerDied","Data":"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"}
Oct 30 15:25:26.264046 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.264023    5064 scope.go:122] "RemoveContainer" containerID="66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"
Oct 30 15:25:26.265889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.265866    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" event={"ID":"f9b560ba-e043-4424-9fc4-962f42f0d321","Type":"ContainerStarted","Data":"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"}
Oct 30 15:25:26.266183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.266167    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:26.266555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.266517    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:26.266632 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.266616    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:26.266680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.266670    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:25:26.373705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:26.373642    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" podStartSLOduration=1.961789689 podStartE2EDuration="2.373625745s" totalImagesPullingTime="411.836056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:26.371203001 +0000 UTC m=+2275.813093732" watchObservedRunningTime="2026-10-30 15:25:26.373625745 +0000 UTC m=+2275.815516460"
Oct 30 15:25:27.275573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:27.275531    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerStarted","Data":"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"}
Oct 30 15:25:28.218121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:28.217248    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:25:28.239861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:28.239081    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 15:25:28.239861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:28.239152    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 15:25:29.237858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:29.234283    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 15:25:29.237858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:29.234352    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 15:25:29.470915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:29.470873    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-g2bcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:25:29.471098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:29.470930    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:25:29.658317 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:25:29.658270    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:25:30.237853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:30.235033    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 15:25:30.237853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:30.235102    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 15:25:31.161946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.161913    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 15:25:31.238704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.238604    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 15:25:31.240030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.239957    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 15:25:31.249522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249493    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249575    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249612    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249677    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249715    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249748    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249844    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249883    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249904    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:31.249937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.249928    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\") pod \"30c1af33-79ab-4202-b9e8-000ab7e22184\" (UID: \"30c1af33-79ab-4202-b9e8-000ab7e22184\") "
Oct 30 15:25:31.250220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.250174    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:31.250810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.250786    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.251725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.251349    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:31.251725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.251383    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:31.251725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.251687    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:25:31.251964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.251867    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.262889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.261269    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:25:31.266368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.266335    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:25:31.266478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.266365    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "kube-api-access-zxlgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:25:31.266478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.266454    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy" pod "30c1af33-79ab-4202-b9e8-000ab7e22184" (UID: "30c1af33-79ab-4202-b9e8-000ab7e22184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:25:31.308525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.308493    5064 generic.go:356] "Generic (PLEG): container finished" podID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerID="37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c" exitCode=0
Oct 30 15:25:31.308696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.308563    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"
Oct 30 15:25:31.308696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.308577    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" event={"ID":"30c1af33-79ab-4202-b9e8-000ab7e22184","Type":"ContainerDied","Data":"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"}
Oct 30 15:25:31.308696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.308665    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr" event={"ID":"30c1af33-79ab-4202-b9e8-000ab7e22184","Type":"ContainerDied","Data":"1d565de192dffe86937f098287e2b6e84e921d3f2c16743ae11156049cf6ce62"}
Oct 30 15:25:31.308696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.308689    5064 scope.go:122] "RemoveContainer" containerID="37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"
Oct 30 15:25:31.360773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360742    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.360773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360769    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.361035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360783    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.361035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360797    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxlgb\" (UniqueName: \"kubernetes.io/projected/30c1af33-79ab-4202-b9e8-000ab7e22184-kube-api-access-zxlgb\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.361035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360807    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.361035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360815    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1af33-79ab-4202-b9e8-000ab7e22184-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.361035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.360824    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1af33-79ab-4202-b9e8-000ab7e22184-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:25:31.363865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.363839    5064 scope.go:122] "RemoveContainer" containerID="37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"
Oct 30 15:25:31.364193 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:25:31.364172    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c\": container with ID starting with 37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c not found: ID does not exist" containerID="37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"
Oct 30 15:25:31.364263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.364209    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c"} err="failed to get container status \"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c\": rpc error: code = NotFound desc = could not find container \"37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c\": container with ID starting with 37fc393e0af2a96829d221a2871304ef58be320b18f21ea1472a4e04b133d64c not found: ID does not exist"
Oct 30 15:25:31.401779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.401749    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 15:25:31.405555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:31.405524    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-g2bcr"]
Oct 30 15:25:32.227374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:32.227339    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 15:25:32.227696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:32.227663    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 15:25:32.285633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:32.282903    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:25:32.438358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:32.437746    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:25:32.725686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:32.725655    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" path="/var/lib/kubelet/pods/30c1af33-79ab-4202-b9e8-000ab7e22184/volumes"
Oct 30 15:25:33.219097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:33.219065    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:25:33.474374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:33.474292    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:25:46.620554 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.620466    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:25:46.621515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.621489    5064 cpu_manager.go:414] "containerMap: removing container" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy"
Oct 30 15:25:46.621515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.621515    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy"
Oct 30 15:25:46.621699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.621684    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c1af33-79ab-4202-b9e8-000ab7e22184" containerName="istio-proxy"
Oct 30 15:25:46.627402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.627377    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.652386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.652351    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:25:46.655214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.654522    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-3f20f9-serving-cert\""
Oct 30 15:25:46.754403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.754367    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.754609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.754448    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrgt\" (UniqueName: \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.754673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.754629    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.754740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.754691    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.856233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.856197    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.856419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.856392    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.856576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.856552    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.856722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.856697    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrgt\" (UniqueName: \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.857583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.857559    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.859102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.859082    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.859437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.859416    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.922897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.917537    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrgt\" (UniqueName: \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\") pod \"maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:46.954888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:46.952719    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:47.305057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:47.302508    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:25:47.587848 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:47.587743    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" event={"ID":"de1735e4-781b-437e-8b5b-3b1ecb784e13","Type":"ContainerStarted","Data":"d59a3c38e763ed10ed603b6cdde902b8721bd0f4cab09e242b3001cb325d4200"}
Oct 30 15:25:48.642322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:48.642278    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" event={"ID":"de1735e4-781b-437e-8b5b-3b1ecb784e13","Type":"ContainerStarted","Data":"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"}
Oct 30 15:25:48.642865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:48.642358    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:48.699337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:48.698324    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" podStartSLOduration=2.314543607 podStartE2EDuration="2.698304658s" totalImagesPullingTime="383.761051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:48.697057994 +0000 UTC m=+2298.138948710" watchObservedRunningTime="2026-10-30 15:25:48.698304658 +0000 UTC m=+2298.140195377"
Oct 30 15:25:54.655934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:25:54.655903    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:25:59.632565 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:25:59.632527    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:26:00.823144 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:00.823105    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:26:00.823680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:00.823375    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" podUID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" containerName="maas-api" containerID="cri-o://efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c" gracePeriod=30
Oct 30 15:26:01.309077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.309042    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:26:01.399621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.399575    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\") pod \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") "
Oct 30 15:26:01.400140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.400110    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\") pod \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") "
Oct 30 15:26:01.400397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.400377    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\") pod \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") "
Oct 30 15:26:01.400614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.400595    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\" (UniqueName: \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\") pod \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\" (UID: \"b241ae0a-71c3-4294-8580-a2d0d42b4b56\") "
Oct 30 15:26:01.402975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.402950    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca" pod "b241ae0a-71c3-4294-8580-a2d0d42b4b56" (UID: "b241ae0a-71c3-4294-8580-a2d0d42b4b56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:26:01.414919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.412423    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt" pod "b241ae0a-71c3-4294-8580-a2d0d42b4b56" (UID: "b241ae0a-71c3-4294-8580-a2d0d42b4b56"). InnerVolumeSpecName "kube-api-access-4bbvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:26:01.439439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.439395    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs" pod "b241ae0a-71c3-4294-8580-a2d0d42b4b56" (UID: "b241ae0a-71c3-4294-8580-a2d0d42b4b56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:26:01.489260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.483047    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls" pod "b241ae0a-71c3-4294-8580-a2d0d42b4b56" (UID: "b241ae0a-71c3-4294-8580-a2d0d42b4b56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:26:01.506590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.506553    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:01.506770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.506609    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b241ae0a-71c3-4294-8580-a2d0d42b4b56-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:01.506770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.506620    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b241ae0a-71c3-4294-8580-a2d0d42b4b56-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:01.506770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.506629    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbvt\" (UniqueName: \"kubernetes.io/projected/b241ae0a-71c3-4294-8580-a2d0d42b4b56-kube-api-access-4bbvt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:01.826474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.826375    5064 generic.go:356] "Generic (PLEG): container finished" podID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" containerID="efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c" exitCode=0
Oct 30 15:26:01.826474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.826429    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"
Oct 30 15:26:01.827033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.826433    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" event={"ID":"b241ae0a-71c3-4294-8580-a2d0d42b4b56","Type":"ContainerDied","Data":"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"}
Oct 30 15:26:01.827033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.826551    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx" event={"ID":"b241ae0a-71c3-4294-8580-a2d0d42b4b56","Type":"ContainerDied","Data":"72b3d778bfeb1b076e955520a87cca77016c1ee70984930bd3943be9e462dd45"}
Oct 30 15:26:01.827033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.826576    5064 scope.go:122] "RemoveContainer" containerID="efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"
Oct 30 15:26:01.838691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.838669    5064 scope.go:122] "RemoveContainer" containerID="efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"
Oct 30 15:26:01.838977 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:01.838960    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c\": container with ID starting with efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c not found: ID does not exist" containerID="efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"
Oct 30 15:26:01.839051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.838986    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c"} err="failed to get container status \"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c\": rpc error: code = NotFound desc = could not find container \"efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c\": container with ID starting with efaef135daac8433ff7f21d2544dfb8897c6325a824079db5a14af89e880a23c not found: ID does not exist"
Oct 30 15:26:01.883897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.883859    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:26:01.899595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:01.899561    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9275a-78b5f475d6-vcqzx"]
Oct 30 15:26:02.725206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:02.725171    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" path="/var/lib/kubelet/pods/b241ae0a-71c3-4294-8580-a2d0d42b4b56/volumes"
Oct 30 15:26:04.736854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:04.729957    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:26:09.454224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:09.454183    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:26:09.454850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:09.454510    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy" containerID="cri-o://047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029" gracePeriod=30
Oct 30 15:26:14.767427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.767392    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:26:14.911722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911691    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.911964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911765    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.911964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.911964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911861    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.911964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911919    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.912230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.911969    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.912230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.912010    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.912230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.912123    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.912230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.912186    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\") pod \"91f1f465-f5da-456f-99e6-3e23bf27e65c\" (UID: \"91f1f465-f5da-456f-99e6-3e23bf27e65c\") "
Oct 30 15:26:14.913669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.913172    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:26:14.913669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.913319    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:26:14.913669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.913634    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:26:14.914383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.914360    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:26:14.916705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.916677    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "kube-api-access-qmkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:26:14.917048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.917023    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:26:14.919035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.918999    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:26:14.919326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.919304    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:26:14.919866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:14.919811    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo" pod "91f1f465-f5da-456f-99e6-3e23bf27e65c" (UID: "91f1f465-f5da-456f-99e6-3e23bf27e65c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:26:15.014651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014564    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014605    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1f465-f5da-456f-99e6-3e23bf27e65c-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014621    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014635    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014652    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkd8\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-kube-api-access-qmkd8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014667    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014680    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014693    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.014994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.014709    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1f465-f5da-456f-99e6-3e23bf27e65c-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:26:15.035603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.035567    5064 generic.go:356] "Generic (PLEG): container finished" podID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerID="047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029" exitCode=0
Oct 30 15:26:15.035789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.035630    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"
Oct 30 15:26:15.035789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.035648    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerDied","Data":"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"}
Oct 30 15:26:15.035789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.035719    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8" event={"ID":"91f1f465-f5da-456f-99e6-3e23bf27e65c","Type":"ContainerDied","Data":"d517ab93e37e7b175d9cd2c452e1552904a3d9e20371d8391f1566753d0092a1"}
Oct 30 15:26:15.035789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.035750    5064 scope.go:122] "RemoveContainer" containerID="047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"
Oct 30 15:26:15.068217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.068187    5064 scope.go:122] "RemoveContainer" containerID="0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"
Oct 30 15:26:15.083416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.083386    5064 scope.go:122] "RemoveContainer" containerID="047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"
Oct 30 15:26:15.083822 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:15.083801    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029\": container with ID starting with 047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029 not found: ID does not exist" containerID="047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"
Oct 30 15:26:15.083898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.083846    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029"} err="failed to get container status \"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029\": rpc error: code = NotFound desc = could not find container \"047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029\": container with ID starting with 047c6fd0d01a11401f48a964b21413d2310c053ed8d6bde1dc9c5af1c9a8e029 not found: ID does not exist"
Oct 30 15:26:15.083898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.083874    5064 scope.go:122] "RemoveContainer" containerID="0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"
Oct 30 15:26:15.084192 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:15.084177    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db\": container with ID starting with 0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db not found: ID does not exist" containerID="0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"
Oct 30 15:26:15.084251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.084197    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db"} err="failed to get container status \"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db\": rpc error: code = NotFound desc = could not find container \"0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db\": container with ID starting with 0cf5d869f3b909539aee4ef688fa69858d09893b2ee46aaa5cba168f4652f2db not found: ID does not exist"
Oct 30 15:26:15.113779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.112184    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:26:15.122886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.119440    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-vxdp8"]
Oct 30 15:26:15.127065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.126036    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.127963    5064 cpu_manager.go:414] "containerMap: removing container" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.127994    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128044    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" containerName="maas-api"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128052    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" containerName="maas-api"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128110    5064 cpu_manager.go:414] "containerMap: removing container" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128118    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128283    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128303    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b241ae0a-71c3-4294-8580-a2d0d42b4b56" containerName="maas-api"
Oct 30 15:26:15.128543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.128322    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" containerName="istio-proxy"
Oct 30 15:26:15.135723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.135627    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.170842 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:15.152396    5064 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f1f465_f5da_456f_99e6_3e23bf27e65c.slice/crio-d517ab93e37e7b175d9cd2c452e1552904a3d9e20371d8391f1566753d0092a1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f1f465_f5da_456f_99e6_3e23bf27e65c.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 15:26:15.170842 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.164016    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7024f9-openshift-default-dockercfg-7m2gr\""
Oct 30 15:26:15.171256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.171232    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:26:15.218033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.217995    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218051    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218219    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218362    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218486    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218523    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnb7\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218562    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218704    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.218781 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.218747    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.327823    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.327916    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnb7\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.327965    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328051    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328080    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328177    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328222    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328303    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.333490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.328383    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.336962    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.337407    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.337708    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.337747    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.338030    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.343014    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.349108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.344644    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.395109 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.395069    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.399866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.398490    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnb7\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.503910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.494688    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:15.936996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.936964    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:26:15.938970 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:26:15.938941    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764bb822_4d34_40e2_aab0_b0b5933faf56.slice/crio-da0ae635826e61026c5631479cc800b08c11cf7574c6b85ba206f75b21524b0f WatchSource:0}: Error finding container da0ae635826e61026c5631479cc800b08c11cf7574c6b85ba206f75b21524b0f: Status 404 returned error can't find the container with id da0ae635826e61026c5631479cc800b08c11cf7574c6b85ba206f75b21524b0f
Oct 30 15:26:15.942154 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.942119    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:15.942223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.942190    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:15.942265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:15.942221    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:16.048859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.048787    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerStarted","Data":"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"}
Oct 30 15:26:16.049065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.048895    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerStarted","Data":"da0ae635826e61026c5631479cc800b08c11cf7574c6b85ba206f75b21524b0f"}
Oct 30 15:26:16.051379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.051344    5064 generic.go:356] "Generic (PLEG): container finished" podID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerID="ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3" exitCode=0
Oct 30 15:26:16.051549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.051445    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerDied","Data":"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"}
Oct 30 15:26:16.052049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.052020    5064 scope.go:122] "RemoveContainer" containerID="ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"
Oct 30 15:26:16.054578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.054533    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:16.054726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.054637    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:16.054726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.054685    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:16.111278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.111230    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podStartSLOduration=1.111216181 podStartE2EDuration="1.111216181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:16.110971618 +0000 UTC m=+2325.552862334" watchObservedRunningTime="2026-10-30 15:26:16.111216181 +0000 UTC m=+2325.553106897"
Oct 30 15:26:16.528881 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.528813    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:16.724807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:16.724768    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f1f465-f5da-456f-99e6-3e23bf27e65c" path="/var/lib/kubelet/pods/91f1f465-f5da-456f-99e6-3e23bf27e65c/volumes"
Oct 30 15:26:17.061601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:17.061559    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerStarted","Data":"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"}
Oct 30 15:26:17.551439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:17.551403    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:18.010905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:18.010868    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:26:18.011199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:18.011176    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 15:26:18.011253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:18.011230    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 15:26:18.068797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:18.068763    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:18.069868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:18.069842    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:26:19.011590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:19.011554    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 15:26:19.011770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:19.011623    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 15:26:20.011117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:20.011077    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 15:26:20.011552 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:20.011143    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 15:26:21.011114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:21.011070    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 15:26:21.011308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:21.011142    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 15:26:22.011426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:22.011336    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 15:26:22.011426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:22.011406    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 15:26:23.016580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:23.016545    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:26:23.017116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:23.016858    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:26:23.017712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:23.017693    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:26:28.793887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.793803    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:26:28.799358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.799332    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.838987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.838944    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:26:28.884822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.884767    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.884992    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885099    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885192 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885149    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885367    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885426    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885478    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885553    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxc8\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.885646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.885593    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987071    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987221    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987254    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987355    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987394    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.987488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987438    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987505    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987511    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxc8\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987571    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987593    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987660    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987789    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.987928    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.988449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.988250    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.990236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.990214    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:28.990327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:28.990239    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:29.012973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.012936    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:29.013119 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.013048    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxc8\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:29.115890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.115759    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:29.449992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.449965    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:26:29.451099 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:26:29.451074    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e449a94_3bb9_4085_b5b6_2517018d8ac1.slice/crio-14fac4133f4f3fe5b6d2aeeea89f4e0b9a6fca361c260ec148443d6f7fb83b1a WatchSource:0}: Error finding container 14fac4133f4f3fe5b6d2aeeea89f4e0b9a6fca361c260ec148443d6f7fb83b1a: Status 404 returned error can't find the container with id 14fac4133f4f3fe5b6d2aeeea89f4e0b9a6fca361c260ec148443d6f7fb83b1a
Oct 30 15:26:29.454309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.454273    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:29.454412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.454354    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:29.454472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:29.454387    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:26:29.640597 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:29.640559    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:26:30.181854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:30.181561    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" event={"ID":"8e449a94-3bb9-4085-b5b6-2517018d8ac1","Type":"ContainerStarted","Data":"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"}
Oct 30 15:26:30.181854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:30.181688    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" event={"ID":"8e449a94-3bb9-4085-b5b6-2517018d8ac1","Type":"ContainerStarted","Data":"14fac4133f4f3fe5b6d2aeeea89f4e0b9a6fca361c260ec148443d6f7fb83b1a"}
Oct 30 15:26:30.268898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:30.268416    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podStartSLOduration=2.2683909780000002 podStartE2EDuration="2.268390978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:30.248878481 +0000 UTC m=+2339.690769198" watchObservedRunningTime="2026-10-30 15:26:30.268390978 +0000 UTC m=+2339.710281696"
Oct 30 15:26:31.131940 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:31.131901    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:31.138019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:31.137954    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:31.138203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:31.138058    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:32.136853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:32.136328    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:32.136853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:32.136413    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:33.137908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:33.127265    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:33.137908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:33.127355    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:34.139853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:34.139307    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:34.139853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:34.139376    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:35.140863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:35.137661    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:35.140863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:35.137736    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:36.116960 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:36.116912    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 15:26:36.117173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:36.116998    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 15:26:37.121891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:37.121853    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:37.122425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:37.122400    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:37.123339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:37.123315    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:26:47.089188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:47.089147    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:26:50.897854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:50.892675    5064 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:26:51.362910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:26:51.361818    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 15:26:59.634693 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:26:59.634652    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:27:01.890649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.890607    5064 generic.go:356] "Generic (PLEG): container finished" podID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerID="3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb" exitCode=0
Oct 30 15:27:01.891153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.890655    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerDied","Data":"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"}
Oct 30 15:27:01.891296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.891274    5064 scope.go:122] "RemoveContainer" containerID="3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"
Oct 30 15:27:01.893587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.893553    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:27:01.893707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.893647    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:27:01.893707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:01.893700    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:27:02.121370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.121188    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" podUID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" containerName="main" containerID="cri-o://515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b" gracePeriod=45
Oct 30 15:27:02.520654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.520623    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:27:02.641454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641407    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.641667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641595    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.641744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641663    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\" (UniqueName: \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.641744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641735    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.641887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641853    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.641887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.641875    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\") pod \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\" (UID: \"4fa7a30f-c95f-44df-8739-bdf46e7f3a23\") "
Oct 30 15:27:02.646594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.646058    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:02.647137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.647060    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:02.647137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.647101    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:02.649770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.649734    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:27:02.649943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.649773    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "kube-api-access-vppbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:02.649943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.649895    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm" pod "4fa7a30f-c95f-44df-8739-bdf46e7f3a23" (UID: "4fa7a30f-c95f-44df-8739-bdf46e7f3a23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:02.743812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743779    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.743812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743810    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.743812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743821    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppbl\" (UniqueName: \"kubernetes.io/projected/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-kube-api-access-vppbl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.743812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743848    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.743812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743863    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.744165 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.743874    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa7a30f-c95f-44df-8739-bdf46e7f3a23-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:02.869716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.869678    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:27:02.900381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.900291    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerStarted","Data":"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"}
Oct 30 15:27:02.901817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.901787    5064 generic.go:356] "Generic (PLEG): container finished" podID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" containerID="515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b" exitCode=0
Oct 30 15:27:02.901961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.901861    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"
Oct 30 15:27:02.901961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.901858    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" event={"ID":"4fa7a30f-c95f-44df-8739-bdf46e7f3a23","Type":"ContainerDied","Data":"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"}
Oct 30 15:27:02.902044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.901976    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp" event={"ID":"4fa7a30f-c95f-44df-8739-bdf46e7f3a23","Type":"ContainerDied","Data":"6401c23882c7e48bbd33b4b87c25efe8c619fea1facfcb408fbe3053419b0fee"}
Oct 30 15:27:02.902044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.902001    5064 scope.go:122] "RemoveContainer" containerID="515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"
Oct 30 15:27:02.939201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.939054    5064 scope.go:122] "RemoveContainer" containerID="515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"
Oct 30 15:27:02.939513 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:02.939486    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b\": container with ID starting with 515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b not found: ID does not exist" containerID="515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"
Oct 30 15:27:02.939594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:02.939521    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b"} err="failed to get container status \"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b\": rpc error: code = NotFound desc = could not find container \"515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b\": container with ID starting with 515552000e465d93f6db2510c6c74439617db882763f4cb023a80651eb6ff29b not found: ID does not exist"
Oct 30 15:27:03.012604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:03.012569    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:27:03.050856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:03.049300    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-859a8d/test-model-859a8d-kserve-c9f685b8f-b5lqp"]
Oct 30 15:27:03.643864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:03.642375    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:27:03.646860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:03.645013    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 15:27:03.646860 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:03.645091    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 15:27:04.652784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:04.649264    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 15:27:04.652784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:04.649335    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 15:27:04.747823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:04.747781    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" path="/var/lib/kubelet/pods/4fa7a30f-c95f-44df-8739-bdf46e7f3a23/volumes"
Oct 30 15:27:04.844896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:04.844244    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:27:04.844896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:04.844498    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" containerID="cri-o://261c42fdd32737731bc46bbb912aafc756cc89f6deb5435d52bc7561415c66a6" gracePeriod=30
Oct 30 15:27:05.641891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:05.641849    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 15:27:05.642108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:05.641903    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 15:27:06.642438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:06.642401    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 15:27:06.643004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:06.642470    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 15:27:07.644322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:07.644271    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 15:27:07.648536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:07.648500    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 15:27:08.646665 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:08.646632    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:27:08.647176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:08.647149    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:27:08.648260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:08.648231    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:27:09.422160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:09.422120    5064 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:27:09.422347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:09.422194    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:27:10.082001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.081966    5064 generic.go:356] "Generic (PLEG): container finished" podID="decca84f-ccda-4c77-978f-b7568ad38457" containerID="261c42fdd32737731bc46bbb912aafc756cc89f6deb5435d52bc7561415c66a6" exitCode=0
Oct 30 15:27:10.082977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.082921    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" event={"ID":"decca84f-ccda-4c77-978f-b7568ad38457","Type":"ContainerDied","Data":"261c42fdd32737731bc46bbb912aafc756cc89f6deb5435d52bc7561415c66a6"}
Oct 30 15:27:10.148269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.148243    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:27:10.227823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.227785    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.227855    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.227932    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.227967    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228050    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228097    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228165    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228168    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:10.228414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228250    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228360    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\") pod \"decca84f-ccda-4c77-978f-b7568ad38457\" (UID: \"decca84f-ccda-4c77-978f-b7568ad38457\") "
Oct 30 15:27:10.228565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.228540    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:27:10.229101 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.229057    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:10.229272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.229097    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:10.229376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.229256    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.229560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.229471    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decca84f-ccda-4c77-978f-b7568ad38457-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.229713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.229679    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:10.231816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.231770    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:10.231975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.231934    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:10.232277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.232252    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:27:10.232373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.232359    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl" pod "decca84f-ccda-4c77-978f-b7568ad38457" (UID: "decca84f-ccda-4c77-978f-b7568ad38457"). InnerVolumeSpecName "kube-api-access-g8hzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:10.331411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331375    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decca84f-ccda-4c77-978f-b7568ad38457-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331405    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331415    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331423    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331432    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331441    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decca84f-ccda-4c77-978f-b7568ad38457-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:10.331660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:10.331450    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hzl\" (UniqueName: \"kubernetes.io/projected/decca84f-ccda-4c77-978f-b7568ad38457-kube-api-access-g8hzl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:11.091759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:11.091656    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"
Oct 30 15:27:11.091759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:11.091689    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m" event={"ID":"decca84f-ccda-4c77-978f-b7568ad38457","Type":"ContainerDied","Data":"67c88c6c723ba96b0a8cd6f64fd99eb247e9f9f59fe95d758600faac15ba5652"}
Oct 30 15:27:11.092234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:11.091763    5064 scope.go:122] "RemoveContainer" containerID="261c42fdd32737731bc46bbb912aafc756cc89f6deb5435d52bc7561415c66a6"
Oct 30 15:27:11.162913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:11.162873    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:27:11.176030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:11.175999    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-mfv4m"]
Oct 30 15:27:12.725170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:12.725128    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decca84f-ccda-4c77-978f-b7568ad38457" path="/var/lib/kubelet/pods/decca84f-ccda-4c77-978f-b7568ad38457/volumes"
Oct 30 15:27:14.501954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:14.501912    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:27:25.909780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.909742    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:27:25.910542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910525    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" containerName="main"
Oct 30 15:27:25.910594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910542    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" containerName="main"
Oct 30 15:27:25.910594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910560    5064 cpu_manager.go:414] "containerMap: removing container" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy"
Oct 30 15:27:25.910594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910565    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy"
Oct 30 15:27:25.910688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910678    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa7a30f-c95f-44df-8739-bdf46e7f3a23" containerName="main"
Oct 30 15:27:25.910688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.910687    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="decca84f-ccda-4c77-978f-b7568ad38457" containerName="istio-proxy"
Oct 30 15:27:25.914381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.914351    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:25.962853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.956818    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-5a5948-serving-cert\""
Oct 30 15:27:25.979854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:25.967913    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:27:26.134943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.110760    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk95\" (UniqueName: \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.134943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.120022    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.134943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.120206    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.134943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.120381    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.381856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.381802    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.382087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.381945    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk95\" (UniqueName: \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.382087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.382011    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.382207 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.382096    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.382292 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:26.382251    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-serving-cert: secret "maas-api-e2e-shared-b-w4-5a5948-serving-cert" not found
Oct 30 15:27:26.382347 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:26.382338    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls podName:9ba81452-32ec-4672-9892-2dbbabe73285 nodeName:}" failed. No retries permitted until 2026-10-30 15:27:26.882319975 +0000 UTC m=+2396.324210669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls") pod "maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" (UID: "9ba81452-32ec-4672-9892-2dbbabe73285") : secret "maas-api-e2e-shared-b-w4-5a5948-serving-cert" not found
Oct 30 15:27:26.382724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.382704    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.384487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.384458    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.409612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.409577    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk95\" (UniqueName: \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.909760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.890059    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:26.915520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:26.915436    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:27.203741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:27.203591    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:27.533075 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:27:27.533041    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba81452_32ec_4672_9892_2dbbabe73285.slice/crio-67ebe50941fdc027bb94b5900c7f5b8dbf1be1725a01a092fe9c0bfb16e6bd47 WatchSource:0}: Error finding container 67ebe50941fdc027bb94b5900c7f5b8dbf1be1725a01a092fe9c0bfb16e6bd47: Status 404 returned error can't find the container with id 67ebe50941fdc027bb94b5900c7f5b8dbf1be1725a01a092fe9c0bfb16e6bd47
Oct 30 15:27:27.534109 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:27.534078    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:27:27.534783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:27.534762    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:27:28.320686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:28.320495    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" event={"ID":"9ba81452-32ec-4672-9892-2dbbabe73285","Type":"ContainerStarted","Data":"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"}
Oct 30 15:27:28.320686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:28.320568    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" event={"ID":"9ba81452-32ec-4672-9892-2dbbabe73285","Type":"ContainerStarted","Data":"67ebe50941fdc027bb94b5900c7f5b8dbf1be1725a01a092fe9c0bfb16e6bd47"}
Oct 30 15:27:28.320686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:28.320599    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:29.528709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.528664    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" podUID="d028e8f6-d6d7-4e07-980e-b2d587618020" containerName="main" containerID="cri-o://05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75" gracePeriod=45
Oct 30 15:27:29.641100 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:29.641058    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:27:29.838584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.838556    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:27:29.902050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.901978    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" podStartSLOduration=4.481500565 podStartE2EDuration="4.901955092s" totalImagesPullingTime="420.454527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:28.375611734 +0000 UTC m=+2397.817502453" watchObservedRunningTime="2026-10-30 15:27:29.901955092 +0000 UTC m=+2399.343845811"
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903259    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\" (UniqueName: \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903323    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903423    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903495    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903630    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.903694    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\" (UniqueName: \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\") pod \"d028e8f6-d6d7-4e07-980e-b2d587618020\" (UID: \"d028e8f6-d6d7-4e07-980e-b2d587618020\") "
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.904095    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.904436    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:29.904503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.904461    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:29.905133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.904748    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:29.906942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.906595    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:29.909406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.909062    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "kube-api-access-m2h7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:29.910481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:29.910455    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs" pod "d028e8f6-d6d7-4e07-980e-b2d587618020" (UID: "d028e8f6-d6d7-4e07-980e-b2d587618020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:27:30.006110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.006061    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2h7j\" (UniqueName: \"kubernetes.io/projected/d028e8f6-d6d7-4e07-980e-b2d587618020-kube-api-access-m2h7j\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:30.006308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.006120    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d028e8f6-d6d7-4e07-980e-b2d587618020-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:30.006308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.006137    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:30.006308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.006149    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:30.006308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.006162    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d028e8f6-d6d7-4e07-980e-b2d587618020-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:30.338254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.338221    5064 generic.go:356] "Generic (PLEG): container finished" podID="d028e8f6-d6d7-4e07-980e-b2d587618020" containerID="05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75" exitCode=0
Oct 30 15:27:30.338444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.338314    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"
Oct 30 15:27:30.338444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.338316    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" event={"ID":"d028e8f6-d6d7-4e07-980e-b2d587618020","Type":"ContainerDied","Data":"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"}
Oct 30 15:27:30.338444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.338368    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc" event={"ID":"d028e8f6-d6d7-4e07-980e-b2d587618020","Type":"ContainerDied","Data":"db483c5a3e4c70e4cc1be24584b206d8b0c09a30ec9a26e4842450f7688b8692"}
Oct 30 15:27:30.338444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.338387    5064 scope.go:122] "RemoveContainer" containerID="05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"
Oct 30 15:27:30.351747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.351720    5064 scope.go:122] "RemoveContainer" containerID="05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"
Oct 30 15:27:30.352224 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:30.352196    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75\": container with ID starting with 05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75 not found: ID does not exist" containerID="05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"
Oct 30 15:27:30.352342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.352229    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75"} err="failed to get container status \"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75\": rpc error: code = NotFound desc = could not find container \"05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75\": container with ID starting with 05b3462680a8e371a082ff577fbe93d750d029f5862f66a81f910e7bc983cc75 not found: ID does not exist"
Oct 30 15:27:30.443678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.443634    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:27:30.447686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.447655    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-13009a/test-model-13009a-kserve-5d6ddbbc9f-6m4rc"]
Oct 30 15:27:30.726427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:30.726396    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d028e8f6-d6d7-4e07-980e-b2d587618020" path="/var/lib/kubelet/pods/d028e8f6-d6d7-4e07-980e-b2d587618020/volumes"
Oct 30 15:27:31.211351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:31.211315    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:27:31.212329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:31.212306    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:27:34.336067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:34.336027    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:27:45.556719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:45.556676    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:27:45.557267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:45.556959    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy" containerID="cri-o://f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774" gracePeriod=30
Oct 30 15:27:50.920559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:50.920523    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:27:51.018732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.018694    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.018956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.018747    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.018956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.018909    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019050    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019078    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019117    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019121    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:51.019216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019201    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:51.019390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019226    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019280    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.019390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.019330    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\") pod \"18362fdf-512b-4462-81bb-f3733400fb17\" (UID: \"18362fdf-512b-4462-81bb-f3733400fb17\") "
Oct 30 15:27:51.020098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.020069    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:27:51.020220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.020118    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.020220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.020138    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.020220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.020206    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:51.020415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.020313    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:51.021721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.021691    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:27:51.021851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.021774    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:27:51.022597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.022570    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "kube-api-access-7v8fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:51.022676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.022604    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token" pod "18362fdf-512b-4462-81bb-f3733400fb17" (UID: "18362fdf-512b-4462-81bb-f3733400fb17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:27:51.121863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121820    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18362fdf-512b-4462-81bb-f3733400fb17-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.121863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121865    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.122072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121877    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8fx\" (UniqueName: \"kubernetes.io/projected/18362fdf-512b-4462-81bb-f3733400fb17-kube-api-access-7v8fx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.122072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121887    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.122072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121897    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18362fdf-512b-4462-81bb-f3733400fb17-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.122072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121907    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.122072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.121916    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18362fdf-512b-4462-81bb-f3733400fb17-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:27:51.520218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.520119    5064 generic.go:356] "Generic (PLEG): container finished" podID="18362fdf-512b-4462-81bb-f3733400fb17" containerID="f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774" exitCode=0
Oct 30 15:27:51.520218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.520171    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"
Oct 30 15:27:51.520218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.520206    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerDied","Data":"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"}
Oct 30 15:27:51.520483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.520265    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg" event={"ID":"18362fdf-512b-4462-81bb-f3733400fb17","Type":"ContainerDied","Data":"8f0e363780e64bc8458dd199f83779c9348125d3baad5e1193cd0e20080db8ce"}
Oct 30 15:27:51.520483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.520290    5064 scope.go:122] "RemoveContainer" containerID="f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"
Oct 30 15:27:51.532546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.532520    5064 scope.go:122] "RemoveContainer" containerID="c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"
Oct 30 15:27:51.542806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.542785    5064 scope.go:122] "RemoveContainer" containerID="f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"
Oct 30 15:27:51.543101 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:51.543080    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774\": container with ID starting with f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774 not found: ID does not exist" containerID="f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"
Oct 30 15:27:51.543179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.543115    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774"} err="failed to get container status \"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774\": rpc error: code = NotFound desc = could not find container \"f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774\": container with ID starting with f8b1c821ee5460c4a64805c3c1c1597fd67d33054287d2c947e87491d97be774 not found: ID does not exist"
Oct 30 15:27:51.543179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.543145    5064 scope.go:122] "RemoveContainer" containerID="c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"
Oct 30 15:27:51.543405 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:51.543386    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081\": container with ID starting with c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081 not found: ID does not exist" containerID="c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"
Oct 30 15:27:51.543446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.543413    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081"} err="failed to get container status \"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081\": rpc error: code = NotFound desc = could not find container \"c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081\": container with ID starting with c78453aced1f19ad7377100419786cdea62858e708e1ccde044d6e67f129d081 not found: ID does not exist"
Oct 30 15:27:51.619115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.619024    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:27:51.624983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:51.624946    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-859a8d-openshift-default-6d5b7976bf-zhfvg"]
Oct 30 15:27:52.725023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:52.724991    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18362fdf-512b-4462-81bb-f3733400fb17" path="/var/lib/kubelet/pods/18362fdf-512b-4462-81bb-f3733400fb17/volumes"
Oct 30 15:27:59.562363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.562326    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:27:59.563140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563114    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d028e8f6-d6d7-4e07-980e-b2d587618020" containerName="main"
Oct 30 15:27:59.563140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563135    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d028e8f6-d6d7-4e07-980e-b2d587618020" containerName="main"
Oct 30 15:27:59.563348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563159    5064 cpu_manager.go:414] "containerMap: removing container" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:27:59.563348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563167    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:27:59.563463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563391    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:27:59.563463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563416    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:27:59.563463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.563437    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d028e8f6-d6d7-4e07-980e-b2d587618020" containerName="main"
Oct 30 15:27:59.567649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.567628    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.592679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.592641    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:27:59.598090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.598056    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7aede6-openshift-default-dockercfg-ksdkq\""
Oct 30 15:27:59.642308 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:27:59.642264    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:27:59.681736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.681692    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.681808    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.681980    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682050    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682195    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4785m\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682232    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682387    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682448    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.682497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.682485    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788418    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788540    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788588    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788618    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788804    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4785m\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788873    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.788976    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.801605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.789032    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.826911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.789080    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.850853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.845356    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.850853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.845714    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.854301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.851159    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.854301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.852794    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.854301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.853375    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.870425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.870218    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.878420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.875418    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.889923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.881220    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:27:59.915189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:27:59.905536    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4785m\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\") pod \"e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") " pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:00.189518 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.189475    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:00.526070 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:28:00.526031    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069d75fc_c250_45a6_8120_7157854e35d1.slice/crio-0f0ff49b83acbd9012810df453adbd768f9c3abe97ddf93a2ee717f5aa516cc8 WatchSource:0}: Error finding container 0f0ff49b83acbd9012810df453adbd768f9c3abe97ddf93a2ee717f5aa516cc8: Status 404 returned error can't find the container with id 0f0ff49b83acbd9012810df453adbd768f9c3abe97ddf93a2ee717f5aa516cc8
Oct 30 15:28:00.526230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.526170    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:28:00.529315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.529281    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:00.529429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.529374    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:00.529429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.529403    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:00.634954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.634913    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerStarted","Data":"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"}
Oct 30 15:28:00.635393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.634978    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerStarted","Data":"0f0ff49b83acbd9012810df453adbd768f9c3abe97ddf93a2ee717f5aa516cc8"}
Oct 30 15:28:00.685356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:00.685301    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podStartSLOduration=1.685285132 podStartE2EDuration="1.685285132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:00.682934297 +0000 UTC m=+2430.124825013" watchObservedRunningTime="2026-10-30 15:28:00.685285132 +0000 UTC m=+2430.127175921"
Oct 30 15:28:01.190051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:01.190015    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:01.196530 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:01.196502    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:01.642801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:01.642768    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:01.643979 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:01.643956    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:28:17.902291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.902257    5064 generic.go:356] "Generic (PLEG): container finished" podID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerID="e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608" exitCode=0
Oct 30 15:28:17.902759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.902337    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerDied","Data":"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"}
Oct 30 15:28:17.902865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.902824    5064 scope.go:122] "RemoveContainer" containerID="e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"
Oct 30 15:28:17.904800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.904762    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:17.904944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.904884    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:17.905011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:17.904940    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:18.069205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:18.069165    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:28:18.952968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:18.945891    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerStarted","Data":"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"}
Oct 30 15:28:19.530858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:19.529541    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:28:19.538065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:19.538010    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:19.538287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:19.538086    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:19.972535 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:19.972495    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:28:19.973079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:19.972815    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" containerID="cri-o://c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc" gracePeriod=30
Oct 30 15:28:20.538025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:20.537987    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:20.538333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:20.538303    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:21.533855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:21.529351    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:21.533855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:21.529429    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:22.537980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:22.537934    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:22.538502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:22.538045    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:23.534856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:23.531293    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:23.534856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:23.531382    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:23.965254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:23.965221    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:28:23.966050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:23.966015    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" podUID="486820ad-d290-4ec4-aebc-ebaf2fc41112" containerName="maas-api" containerID="cri-o://ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063" gracePeriod=30
Oct 30 15:28:24.585115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.538431    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 15:28:24.585115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.538494    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 15:28:24.588494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.587885    5064 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:28:24.588494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.587960    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:28:24.645156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.644742    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:28:24.790634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.790527    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\") pod \"486820ad-d290-4ec4-aebc-ebaf2fc41112\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") "
Oct 30 15:28:24.790634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.790595    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\") pod \"486820ad-d290-4ec4-aebc-ebaf2fc41112\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") "
Oct 30 15:28:24.790925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.790670    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") pod \"486820ad-d290-4ec4-aebc-ebaf2fc41112\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") "
Oct 30 15:28:24.790925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.790737    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\" (UniqueName: \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\") pod \"486820ad-d290-4ec4-aebc-ebaf2fc41112\" (UID: \"486820ad-d290-4ec4-aebc-ebaf2fc41112\") "
Oct 30 15:28:24.791055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.790952    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca" pod "486820ad-d290-4ec4-aebc-ebaf2fc41112" (UID: "486820ad-d290-4ec4-aebc-ebaf2fc41112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:28:24.791639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.791579    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486820ad-d290-4ec4-aebc-ebaf2fc41112-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:24.793092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.793062    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs" pod "486820ad-d290-4ec4-aebc-ebaf2fc41112" (UID: "486820ad-d290-4ec4-aebc-ebaf2fc41112"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:28:24.793372 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.793348    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls" pod "486820ad-d290-4ec4-aebc-ebaf2fc41112" (UID: "486820ad-d290-4ec4-aebc-ebaf2fc41112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:28:24.793462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.793416    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84" pod "486820ad-d290-4ec4-aebc-ebaf2fc41112" (UID: "486820ad-d290-4ec4-aebc-ebaf2fc41112"). InnerVolumeSpecName "kube-api-access-krj84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:28:24.893280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.893239    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:24.893516 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.893502    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486820ad-d290-4ec4-aebc-ebaf2fc41112-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:24.893697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:24.893670    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krj84\" (UniqueName: \"kubernetes.io/projected/486820ad-d290-4ec4-aebc-ebaf2fc41112-kube-api-access-krj84\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.016528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.016499    5064 generic.go:356] "Generic (PLEG): container finished" podID="486820ad-d290-4ec4-aebc-ebaf2fc41112" containerID="ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063" exitCode=0
Oct 30 15:28:25.017016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.016549    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"
Oct 30 15:28:25.017016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.016586    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" event={"ID":"486820ad-d290-4ec4-aebc-ebaf2fc41112","Type":"ContainerDied","Data":"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"}
Oct 30 15:28:25.017016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.016647    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc" event={"ID":"486820ad-d290-4ec4-aebc-ebaf2fc41112","Type":"ContainerDied","Data":"2d3c2c8fbf4038f33c06e636742660edafc870f7be02b5f1d4c16909318a06e4"}
Oct 30 15:28:25.017016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.016671    5064 scope.go:122] "RemoveContainer" containerID="ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"
Oct 30 15:28:25.089422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.089396    5064 scope.go:122] "RemoveContainer" containerID="ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"
Oct 30 15:28:25.089771 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:28:25.089749    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063\": container with ID starting with ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063 not found: ID does not exist" containerID="ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"
Oct 30 15:28:25.089878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.089779    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063"} err="failed to get container status \"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063\": rpc error: code = NotFound desc = could not find container \"ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063\": container with ID starting with ea383a70556a850a8e7d5d4686161d326b27d6bf25e43ee24b0f225416614063 not found: ID does not exist"
Oct 30 15:28:25.107770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.107739    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:28:25.111674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.111649    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13009a-cbf9f8d4-pbswc"]
Oct 30 15:28:25.323194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.323130    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:28:25.405239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405207    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405255    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405389    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405462    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405507    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405610    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.405711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405635    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:25.405711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405679    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:28:25.405891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405715    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.406007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.405988    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:25.406106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.406075    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:25.406963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.406934    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.407125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.406986    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\") pod \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\" (UID: \"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39\") "
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407754    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407931    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407947    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407958    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407968    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.408211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.407981    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.408645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.408529    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:28:25.409167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.409135    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "kube-api-access-69f89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:28:25.409286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.409261    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:25.409366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.409291    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token" pod "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" (UID: "5f7c6af9-cd63-4e58-87b4-f7fa57f10c39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:28:25.509362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.509318    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.509362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.509353    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69f89\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-kube-api-access-69f89\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.509362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.509363    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.509362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.509372    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:25.533520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.533491    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:28:25.533956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.533923    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:28:25.534556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:25.534539    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:28:26.025113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.025078    5064 generic.go:356] "Generic (PLEG): container finished" podID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerID="c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc" exitCode=0
Oct 30 15:28:26.025546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.025132    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"
Oct 30 15:28:26.025546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.025158    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" event={"ID":"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39","Type":"ContainerDied","Data":"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"}
Oct 30 15:28:26.025546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.025214    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j" event={"ID":"5f7c6af9-cd63-4e58-87b4-f7fa57f10c39","Type":"ContainerDied","Data":"c30dd6546aaae2bd6a4e18939ffdfec286faa816af7505288d264558b6cf65eb"}
Oct 30 15:28:26.025546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.025237    5064 scope.go:122] "RemoveContainer" containerID="c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"
Oct 30 15:28:26.037544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.037518    5064 scope.go:122] "RemoveContainer" containerID="c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"
Oct 30 15:28:26.037820 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:28:26.037802    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc\": container with ID starting with c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc not found: ID does not exist" containerID="c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"
Oct 30 15:28:26.037820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.037848    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc"} err="failed to get container status \"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc\": rpc error: code = NotFound desc = could not find container \"c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc\": container with ID starting with c434241382cb55e9d8765216a805d9a053667f94c3cdfc269e24f4108e6627cc not found: ID does not exist"
Oct 30 15:28:26.084724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.084691    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:28:26.098488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.098457    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-8878j"]
Oct 30 15:28:26.728560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.728525    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486820ad-d290-4ec4-aebc-ebaf2fc41112" path="/var/lib/kubelet/pods/486820ad-d290-4ec4-aebc-ebaf2fc41112/volumes"
Oct 30 15:28:26.729162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.729137    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" path="/var/lib/kubelet/pods/5f7c6af9-cd63-4e58-87b4-f7fa57f10c39/volumes"
Oct 30 15:28:26.729546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.729525    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:26.730271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730252    5064 cpu_manager.go:414] "containerMap: removing container" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:28:26.730368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730271    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18362fdf-512b-4462-81bb-f3733400fb17" containerName="istio-proxy"
Oct 30 15:28:26.730368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730308    5064 cpu_manager.go:414] "containerMap: removing container" podUID="486820ad-d290-4ec4-aebc-ebaf2fc41112" containerName="maas-api"
Oct 30 15:28:26.730368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730316    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486820ad-d290-4ec4-aebc-ebaf2fc41112" containerName="maas-api"
Oct 30 15:28:26.730368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730327    5064 cpu_manager.go:414] "containerMap: removing container" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy"
Oct 30 15:28:26.730368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730335    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy"
Oct 30 15:28:26.730616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730541    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="486820ad-d290-4ec4-aebc-ebaf2fc41112" containerName="maas-api"
Oct 30 15:28:26.730616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.730559    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7c6af9-cd63-4e58-87b4-f7fa57f10c39" containerName="istio-proxy"
Oct 30 15:28:26.734427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.734405    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.754677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.754646    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:26.762362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.762332    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-13009a-openshift-default-dockercfg-2gpnk\""
Oct 30 15:28:26.828291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828258    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828307    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828323    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828471    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828513    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828673    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828725    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828778    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnk8j\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.828842 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.828812    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.930568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930527    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.930737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930575    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.930737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930594    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.930737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930644    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.930737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930676    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930754    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930980    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.930980    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931070    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnk8j\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931110    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931160    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931248    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931395    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.931464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.931409    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.933107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.933081    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.933212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.933152    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.955187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.955162    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:26.955414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:26.955394    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnk8j\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:27.047569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:27.047482    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:27.428559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:27.428527    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:27.429429 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:28:27.429400    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a999d8d_6c10_4bcf_8cd4_0f9afce6d447.slice/crio-7691f7347259dbe29e76228b092fbe9485c836060ceaeebce8902ea45b9569d6 WatchSource:0}: Error finding container 7691f7347259dbe29e76228b092fbe9485c836060ceaeebce8902ea45b9569d6: Status 404 returned error can't find the container with id 7691f7347259dbe29e76228b092fbe9485c836060ceaeebce8902ea45b9569d6
Oct 30 15:28:27.432927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:27.432884    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:27.433020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:27.432976    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:27.433071 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:27.433020    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:28.051501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:28.051465    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" event={"ID":"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447","Type":"ContainerStarted","Data":"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"}
Oct 30 15:28:28.051939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:28.051529    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" event={"ID":"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447","Type":"ContainerStarted","Data":"7691f7347259dbe29e76228b092fbe9485c836060ceaeebce8902ea45b9569d6"}
Oct 30 15:28:28.113670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:28.113601    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podStartSLOduration=2.113579026 podStartE2EDuration="2.113579026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:28.110677987 +0000 UTC m=+2457.552568719" watchObservedRunningTime="2026-10-30 15:28:28.113579026 +0000 UTC m=+2457.555469754"
Oct 30 15:28:29.048038 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:29.047996    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:29.049630 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:29.049602    5064 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 15:28:29.049771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:29.049651    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 15:28:29.659444 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:28:29.659390    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:28:30.051332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:30.051234    5064 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 15:28:30.051498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:30.051368    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 15:28:31.048809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:31.048768    5064 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 15:28:31.049479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:31.048910    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 15:28:32.049657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:32.049605    5064 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 15:28:32.050262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:32.049699    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 15:28:32.363681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:32.361667    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:32.364143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:32.364112    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy" containerID="cri-o://fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa" gracePeriod=30
Oct 30 15:28:34.774769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.774724    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:28:34.780137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.780110    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.802078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.802045    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:28:34.887456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.887425    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.887653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.887622    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.887805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.887775    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.887974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.887953    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.888031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.888017    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.888160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.888141    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bckx\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.888238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.888199    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.888297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.888238    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.888422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.888402    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990250    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990318    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bckx\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990338    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990358    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990473    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990554    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990657    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990733    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990786    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990817    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990823    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.990955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.990929    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.991173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.991071    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.991456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.991435    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.993184 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.993161    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:34.993326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:34.993304    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:35.046914 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.046032    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:35.050653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.050535    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bckx\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:35.095078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.095043    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:35.422608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.422579    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:28:35.424056 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:28:35.424019    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa26d82_61de_49b1_a280_cd57252bb6ad.slice/crio-62f516c60e56cb2c3c6bab81edd24dd9fe3ce6eb74b64db8bd07f3a2a608ae07 WatchSource:0}: Error finding container 62f516c60e56cb2c3c6bab81edd24dd9fe3ce6eb74b64db8bd07f3a2a608ae07: Status 404 returned error can't find the container with id 62f516c60e56cb2c3c6bab81edd24dd9fe3ce6eb74b64db8bd07f3a2a608ae07
Oct 30 15:28:35.427008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.426959    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:35.427098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.427033    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:35.427098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:35.427067    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:28:36.132893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:36.132846    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" event={"ID":"5aa26d82-61de-49b1-a280-cd57252bb6ad","Type":"ContainerStarted","Data":"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"}
Oct 30 15:28:36.133409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:36.132916    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" event={"ID":"5aa26d82-61de-49b1-a280-cd57252bb6ad","Type":"ContainerStarted","Data":"62f516c60e56cb2c3c6bab81edd24dd9fe3ce6eb74b64db8bd07f3a2a608ae07"}
Oct 30 15:28:36.193081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:36.193020    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podStartSLOduration=2.193007045 podStartE2EDuration="2.193007045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:36.192770971 +0000 UTC m=+2465.634661688" watchObservedRunningTime="2026-10-30 15:28:36.193007045 +0000 UTC m=+2465.634897794"
Oct 30 15:28:37.095613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:37.095580    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:37.097123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:37.097089    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:37.097259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:37.097145    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:38.096383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:38.096344    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:38.096799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:38.096403    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:39.095722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.095679    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:39.095933 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.095763    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:39.499198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.499170    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:39.597931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.597891    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.597943    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598064    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598102    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598191    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598229    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:39.598323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598267    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598407    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:39.598486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598418    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598518    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598565    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\") pod \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\" (UID: \"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447\") "
Oct 30 15:28:39.598711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.598657    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:39.599397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.599025    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:28:39.599397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.599325    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.599397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.599346    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.599397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.599360    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.599397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.599372    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.600106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.600082    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:39.601290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.601261    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "kube-api-access-qnk8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:28:39.602299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.602270    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:28:39.602467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.602278    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:28:39.604085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.604018    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo" pod "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" (UID: "8a999d8d-6c10-4bcf-8cd4-0f9afce6d447"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:28:39.700677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.700641    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.700677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.700671    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.700677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.700680    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.701023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.700692    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnk8j\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-kube-api-access-qnk8j\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:39.701023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:39.700701    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:28:40.096176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.096140    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:40.096402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.096328    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:40.397599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.397565    5064 generic.go:356] "Generic (PLEG): container finished" podID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerID="fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa" exitCode=0
Oct 30 15:28:40.397823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.397799    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" event={"ID":"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447","Type":"ContainerDied","Data":"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"}
Oct 30 15:28:40.397990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.397889    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c" event={"ID":"8a999d8d-6c10-4bcf-8cd4-0f9afce6d447","Type":"ContainerDied","Data":"7691f7347259dbe29e76228b092fbe9485c836060ceaeebce8902ea45b9569d6"}
Oct 30 15:28:40.397990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.397914    5064 scope.go:122] "RemoveContainer" containerID="fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"
Oct 30 15:28:40.398115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.398049    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"
Oct 30 15:28:40.443784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.443742    5064 scope.go:122] "RemoveContainer" containerID="fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"
Oct 30 15:28:40.444475 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:28:40.444393    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa\": container with ID starting with fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa not found: ID does not exist" containerID="fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"
Oct 30 15:28:40.444475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.444431    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa"} err="failed to get container status \"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa\": rpc error: code = NotFound desc = could not find container \"fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa\": container with ID starting with fced45b05d1faac2ffce4c1b9a6d7d34d3ce95c4378eaf6eaa43c5b5ddc6b3aa not found: ID does not exist"
Oct 30 15:28:40.478619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.478585    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:40.482379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.482342    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-r5q7c"]
Oct 30 15:28:40.748069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:40.747984    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" path="/var/lib/kubelet/pods/8a999d8d-6c10-4bcf-8cd4-0f9afce6d447/volumes"
Oct 30 15:28:41.095918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:41.095797    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:41.095918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:41.095876    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:42.095466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:42.095427    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 15:28:42.095987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:42.095488    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 15:28:43.121320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:43.121278    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:43.123791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:43.123763    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:43.131329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:43.131296    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:28:44.584704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.584669    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:28:44.585487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.585470    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy"
Oct 30 15:28:44.585532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.585488    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy"
Oct 30 15:28:44.585607 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.585597    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a999d8d-6c10-4bcf-8cd4-0f9afce6d447" containerName="istio-proxy"
Oct 30 15:28:44.589103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.589083    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.630027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.629980    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:28:44.636963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.636920    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7aede6-serving-cert\""
Oct 30 15:28:44.699389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.699355    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.699605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.699444    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.699605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.699512    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.699727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.699628    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrzs\" (UniqueName: \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.803857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.801142    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrzs\" (UniqueName: \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.803857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.801428    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.803857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.801464    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.803857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.801509    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.818759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.818720    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.822737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.822317    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.834900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.831529    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.876117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.876072    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrzs\" (UniqueName: \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\") pod \"maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:44.906012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:44.905979    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:45.214800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:45.214774    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:28:45.215319 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:28:45.215283    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1bf9bf_1396_4951_b1ee_e28b93e51ae1.slice/crio-6903e3cf98b38cc79701d0284786f3a64a86a0659a02dd27942a2bc77a1bc696 WatchSource:0}: Error finding container 6903e3cf98b38cc79701d0284786f3a64a86a0659a02dd27942a2bc77a1bc696: Status 404 returned error can't find the container with id 6903e3cf98b38cc79701d0284786f3a64a86a0659a02dd27942a2bc77a1bc696
Oct 30 15:28:45.460410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:45.460367    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" event={"ID":"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1","Type":"ContainerStarted","Data":"6903e3cf98b38cc79701d0284786f3a64a86a0659a02dd27942a2bc77a1bc696"}
Oct 30 15:28:46.471053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:46.471011    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" event={"ID":"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1","Type":"ContainerStarted","Data":"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"}
Oct 30 15:28:46.471547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:46.471089    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:46.530131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:46.530081    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" podStartSLOduration=2.177520177 podStartE2EDuration="2.530067605s" totalImagesPullingTime="352.547428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:46.526975684 +0000 UTC m=+2475.968866401" watchObservedRunningTime="2026-10-30 15:28:46.530067605 +0000 UTC m=+2475.971958322"
Oct 30 15:28:48.832762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.832719    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:28:48.837678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.837649    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.858868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.858837    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:28:48.863506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.863481    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-09e8eb\"/\"auth-test-model-09e8eb-kserve-self-signed-certs\""
Oct 30 15:28:48.868450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.868400    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-09e8eb\"/\"default-dockercfg-22mc8\""
Oct 30 15:28:48.868450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.868549    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-09e8eb\"/\"kube-root-ca.crt\""
Oct 30 15:28:48.868450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.868623    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-09e8eb\"/\"openshift-service-ca.crt\""
Oct 30 15:28:48.902721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.902688    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.902935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.902747    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pjl\" (UniqueName: \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.902935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.902910    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.903132 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.903110    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.903290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.903276    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:48.903348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:48.903308    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005235    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005368    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005389    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005433    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005475    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pjl\" (UniqueName: \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005714 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005549    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005924    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.005985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005943    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.006147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.005985    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.008441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.008418    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.010212 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.010191    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.030614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.030579    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pjl\" (UniqueName: \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\") pod \"auth-test-model-09e8eb-kserve-f564fc869-5rncj\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") " pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.160036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.159198    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:49.498138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.498100    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:28:49.498758 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:28:49.498723    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8abae952_eafb_4cc6_b1f9_edd5709d0ff2.slice/crio-333b1a966d06f6024532b88d7e505afa7c96e1f174366686e091365689fed379 WatchSource:0}: Error finding container 333b1a966d06f6024532b88d7e505afa7c96e1f174366686e091365689fed379: Status 404 returned error can't find the container with id 333b1a966d06f6024532b88d7e505afa7c96e1f174366686e091365689fed379
Oct 30 15:28:49.523248 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:49.523216    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" event={"ID":"8abae952-eafb-4cc6-b1f9-edd5709d0ff2","Type":"ContainerStarted","Data":"333b1a966d06f6024532b88d7e505afa7c96e1f174366686e091365689fed379"}
Oct 30 15:28:50.533050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:50.533011    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" event={"ID":"8abae952-eafb-4cc6-b1f9-edd5709d0ff2","Type":"ContainerStarted","Data":"aca3758242b6fe3fa6921e2a64a1c6e4524c4583adf5b5a7135b808c3293bfce"}
Oct 30 15:28:50.578599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:50.578538    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" podStartSLOduration=2.578523594 podStartE2EDuration="2.578523594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:50.576356088 +0000 UTC m=+2480.018246805" watchObservedRunningTime="2026-10-30 15:28:50.578523594 +0000 UTC m=+2480.020414318"
Oct 30 15:28:50.646565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:50.646524    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:28:51.160149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:51.160109    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:52.512683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:52.512638    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:28:57.382412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:57.382376    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:28:59.177594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:59.173273    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:59.216032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:59.215995    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:28:59.632920 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:28:59.632871    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:28:59.643302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:28:59.643278    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:29:05.675773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:05.675719    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" podUID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" containerName="main" containerID="cri-o://1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403" gracePeriod=45
Oct 30 15:29:05.975856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:05.975799    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:29:06.046876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.046802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.047075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.046916    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\" (UniqueName: \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.047075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047068    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047207    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047283    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047350    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047392    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\") pod \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\" (UID: \"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3\") "
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047599    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.047610    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.048143    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.048165    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.048996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.048178    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.050127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.050091    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:06.050217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.050124    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "kube-api-access-54lf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:29:06.050217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.050175    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs" pod "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" (UID: "fd27a4ba-e27d-4bf2-ab28-018cc256c4f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:29:06.150075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.150036    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.150075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.150070    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lf2\" (UniqueName: \"kubernetes.io/projected/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-kube-api-access-54lf2\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.150075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.150081    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:06.801079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.801041    5064 generic.go:356] "Generic (PLEG): container finished" podID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" containerID="1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403" exitCode=0
Oct 30 15:29:06.801561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.801097    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"
Oct 30 15:29:06.801561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.801132    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" event={"ID":"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3","Type":"ContainerDied","Data":"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"}
Oct 30 15:29:06.801561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.801192    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2" event={"ID":"fd27a4ba-e27d-4bf2-ab28-018cc256c4f3","Type":"ContainerDied","Data":"caa191e0508ca14b2344656ce6cacd5726517e1e4da7291be5400fe037d74654"}
Oct 30 15:29:06.801561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.801212    5064 scope.go:122] "RemoveContainer" containerID="1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"
Oct 30 15:29:06.812602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.812585    5064 scope.go:122] "RemoveContainer" containerID="1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"
Oct 30 15:29:06.812935 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:06.812914    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403\": container with ID starting with 1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403 not found: ID does not exist" containerID="1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"
Oct 30 15:29:06.813020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.812942    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403"} err="failed to get container status \"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403\": rpc error: code = NotFound desc = could not find container \"1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403\": container with ID starting with 1ce85817e4825abc018b0b23c050b3c719dff47f3bd406906322913dede31403 not found: ID does not exist"
Oct 30 15:29:06.886945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.886896    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:29:06.892472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:06.892430    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-985bf6/rate-test-model-985bf6-kserve-7b79dc7685-q5gg2"]
Oct 30 15:29:08.725433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:08.725397    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" path="/var/lib/kubelet/pods/fd27a4ba-e27d-4bf2-ab28-018cc256c4f3/volumes"
Oct 30 15:29:09.445088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.445044    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:29:09.445896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.445873    5064 cpu_manager.go:414] "containerMap: removing container" podUID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" containerName="main"
Oct 30 15:29:09.445896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.445896    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" containerName="main"
Oct 30 15:29:09.446141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.446126    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27a4ba-e27d-4bf2-ab28-018cc256c4f3" containerName="main"
Oct 30 15:29:09.451246 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.451225    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.485859 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.484630    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-76473c-openshift-default-dockercfg-p7vs4\""
Oct 30 15:29:09.487424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.486480    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:29:09.609046 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609007    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609111    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609170    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrswp\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609239    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609381    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609504    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609628    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609680    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.609772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.609746    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711354 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711268    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711386    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711405    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711432    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711469    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711515    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711567    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrswp\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711644    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.711820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711707    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.712009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.711895    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.712095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.712033    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.712210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.712182    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.712380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.712359    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.712445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.712319    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.713947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.713923    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.714204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.714183    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.737286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.737254    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.737645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.737431    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrswp\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:09.798605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:09.798570    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:10.178603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.178563    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:29:10.178877 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:29:10.178812    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c684b0_b899_4e29_862d_d190a492ffb7.slice/crio-0822f9390d023d8d00bda50c7e92f3fe137deee48cee74016998b111693a6146 WatchSource:0}: Error finding container 0822f9390d023d8d00bda50c7e92f3fe137deee48cee74016998b111693a6146: Status 404 returned error can't find the container with id 0822f9390d023d8d00bda50c7e92f3fe137deee48cee74016998b111693a6146
Oct 30 15:29:10.182085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.182047    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:10.182175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.182122    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:10.182175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.182155    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:10.836237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.836201    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" event={"ID":"17c684b0-b899-4e29-862d-d190a492ffb7","Type":"ContainerStarted","Data":"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"}
Oct 30 15:29:10.836743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.836265    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" event={"ID":"17c684b0-b899-4e29-862d-d190a492ffb7","Type":"ContainerStarted","Data":"0822f9390d023d8d00bda50c7e92f3fe137deee48cee74016998b111693a6146"}
Oct 30 15:29:10.884597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:10.884537    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podStartSLOduration=1.884511338 podStartE2EDuration="1.884511338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:10.883935311 +0000 UTC m=+2500.325826030" watchObservedRunningTime="2026-10-30 15:29:10.884511338 +0000 UTC m=+2500.326402056"
Oct 30 15:29:11.799010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:11.798975    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:11.800698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:11.800669    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:11.800872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:11.800721    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:12.408196 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.408147    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerName="main" containerID="cri-o://a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6" gracePeriod=45
Oct 30 15:29:12.799819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.799769    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:12.799819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.799878    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:12.811608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.811575    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:29:12.888915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.888868    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.889097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.888987    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\" (UniqueName: \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.889097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.889047    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\" (UniqueName: \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.889206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.889192    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.889273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.889257    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.889413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.889398    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\") pod \"3baf93c8-b179-4a6f-a84b-19f3257fa812\" (UID: \"3baf93c8-b179-4a6f-a84b-19f3257fa812\") "
Oct 30 15:29:12.890503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.890304    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:12.890503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.890420    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:12.890503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.890440    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:12.894460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.894418    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:29:12.899908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.898659    5064 generic.go:356] "Generic (PLEG): container finished" podID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerID="a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6" exitCode=0
Oct 30 15:29:12.899908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.899590    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"
Oct 30 15:29:12.900090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.899900    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" event={"ID":"3baf93c8-b179-4a6f-a84b-19f3257fa812","Type":"ContainerDied","Data":"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"}
Oct 30 15:29:12.900090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.899952    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7" event={"ID":"3baf93c8-b179-4a6f-a84b-19f3257fa812","Type":"ContainerDied","Data":"36ef77a8ed936df567abdd26e2f25f119b4fc49a0adb53e12ce76058a7d8a971"}
Oct 30 15:29:12.900090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.899975    5064 scope.go:122] "RemoveContainer" containerID="a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"
Oct 30 15:29:12.901218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.900954    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:29:12.907348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.907314    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n" pod "3baf93c8-b179-4a6f-a84b-19f3257fa812" (UID: "3baf93c8-b179-4a6f-a84b-19f3257fa812"). InnerVolumeSpecName "kube-api-access-5g52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:29:12.948874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.948743    5064 scope.go:122] "RemoveContainer" containerID="a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"
Oct 30 15:29:12.949263 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:12.949227    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6\": container with ID starting with a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6 not found: ID does not exist" containerID="a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"
Oct 30 15:29:12.949312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:12.949277    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6"} err="failed to get container status \"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6\": rpc error: code = NotFound desc = could not find container \"a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6\": container with ID starting with a5900468b2b94317d2a39facc5376f86d39c9ddc481b4a9bfab240744d5be0d6 not found: ID does not exist"
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059587    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g52n\" (UniqueName: \"kubernetes.io/projected/3baf93c8-b179-4a6f-a84b-19f3257fa812-kube-api-access-5g52n\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059629    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3baf93c8-b179-4a6f-a84b-19f3257fa812-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059644    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059654    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059665    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.059907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.059675    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3baf93c8-b179-4a6f-a84b-19f3257fa812-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:29:13.260743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.260711    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:29:13.274425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.274396    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-609709/rate-test-model-609709-kserve-5c79f98d45-98wb7"]
Oct 30 15:29:13.799780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.799731    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:13.800268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:13.799802    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:14.747308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:14.747266    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" path="/var/lib/kubelet/pods/3baf93c8-b179-4a6f-a84b-19f3257fa812/volumes"
Oct 30 15:29:14.799239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:14.799197    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:14.799434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:14.799257    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:15.807852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:15.807705    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:15.807852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:15.807768    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:16.823941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:16.823023    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:16.823941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:16.823088    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:17.371360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:17.369136    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:29:17.799240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:17.799143    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 15:29:17.799520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:17.799486    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 15:29:18.361614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.361577    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:29:18.362363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.362347    5064 cpu_manager.go:414] "containerMap: removing container" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerName="main"
Oct 30 15:29:18.362363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.362363    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerName="main"
Oct 30 15:29:18.362499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.362469    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baf93c8-b179-4a6f-a84b-19f3257fa812" containerName="main"
Oct 30 15:29:18.368236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.368212    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.372625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.372595    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.372761 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.372714    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.372959 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.372940    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.373053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.373030    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.387341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.387236    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:29:18.403787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.403747    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b236a6-serving-cert\""
Oct 30 15:29:18.474623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.474587    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.474822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.474645    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.474822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.474745    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.474822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.474785    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.476260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.476235    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.476423 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:18.476406    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-serving-cert: secret "maas-api-e2e-shared-a-w1-b236a6-serving-cert" not found
Oct 30 15:29:18.476495 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:18.476457    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls podName:25073711-e846-4908-93fb-2cf78b5bd4d5 nodeName:}" failed. No retries permitted until 2026-10-30 15:29:18.976444035 +0000 UTC m=+2508.418334729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls") pod "maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" (UID: "25073711-e846-4908-93fb-2cf78b5bd4d5") : secret "maas-api-e2e-shared-a-w1-b236a6-serving-cert" not found
Oct 30 15:29:18.478759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.478734    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.501866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.501814    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:18.556538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.556476    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:29:18.586372 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.586335    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:29:18.586699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.586678    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858150 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858100    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858241    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858304    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858348    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858574    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858685 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858669    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858705    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctl8q\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858863    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.858957 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.858909    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.960894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.960735    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.960922    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961096 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.960958    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctl8q\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.961118    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.961155    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.961154    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.961331    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.961517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.961448    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.962060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.962032    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.962247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.962222    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.962401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.962264    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.962772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.962747    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.963223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.963161    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.963356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.963335    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.964433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.964408    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.965311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.965287    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.986581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.986548    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctl8q\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:18.988639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:18.988609    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:19.041961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.041219    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:19.047855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.042936    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:19.049309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.049271    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:29:19.056475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.056432    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:19.064544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.064501    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:19.067100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.067068    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:19.367641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.367607    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:19.447856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.447221    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:29:19.448723 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:29:19.448700    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ad6564_7674_4295_b749_17cbcde1faf1.slice/crio-a51d7f66f1ae3e5163dd8d1123c062e382a5d2ce758de3bb381c9d13bba0821b WatchSource:0}: Error finding container a51d7f66f1ae3e5163dd8d1123c062e382a5d2ce758de3bb381c9d13bba0821b: Status 404 returned error can't find the container with id a51d7f66f1ae3e5163dd8d1123c062e382a5d2ce758de3bb381c9d13bba0821b
Oct 30 15:29:19.452634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.452236    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:19.452634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.452310    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:19.452634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.452342    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:19.744322 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:19.744295    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:29:20.080927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:20.080400    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" event={"ID":"25073711-e846-4908-93fb-2cf78b5bd4d5","Type":"ContainerStarted","Data":"de3f09be5a402f5bac2faff167febf4a7c570d23c0b9d8a206da92909adfc4c6"}
Oct 30 15:29:20.082583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:20.082546    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" event={"ID":"e0ad6564-7674-4295-b749-17cbcde1faf1","Type":"ContainerStarted","Data":"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"}
Oct 30 15:29:20.082733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:20.082605    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" event={"ID":"e0ad6564-7674-4295-b749-17cbcde1faf1","Type":"ContainerStarted","Data":"a51d7f66f1ae3e5163dd8d1123c062e382a5d2ce758de3bb381c9d13bba0821b"}
Oct 30 15:29:20.139485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:20.139412    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podStartSLOduration=2.139392415 podStartE2EDuration="2.139392415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:20.136352 +0000 UTC m=+2509.578242720" watchObservedRunningTime="2026-10-30 15:29:20.139392415 +0000 UTC m=+2509.581283136"
Oct 30 15:29:21.060856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.060274    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:21.065006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.064966    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:21.065178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.065034    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:21.093073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.093041    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" event={"ID":"25073711-e846-4908-93fb-2cf78b5bd4d5","Type":"ContainerStarted","Data":"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"}
Oct 30 15:29:21.093241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.093106    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:21.167853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:21.167740    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" podStartSLOduration=2.768925311 podStartE2EDuration="3.167724653s" totalImagesPullingTime="398.799342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:21.165080445 +0000 UTC m=+2510.606971164" watchObservedRunningTime="2026-10-30 15:29:21.167724653 +0000 UTC m=+2510.609615396"
Oct 30 15:29:22.062392 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:22.062280    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:22.062392 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:22.062355    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:23.057813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:23.057782    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:23.058027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:23.057846    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:24.056930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:24.056889    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:24.057443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:24.056953    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:25.057384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:25.057348    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:25.057806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:25.057409    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:26.057326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:26.057293    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 15:29:26.057509 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:26.057349    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 15:29:27.062183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:27.062149    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:27.062689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:27.062658    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:27.063646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:27.063626    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:29:28.114928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.114811    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:29:28.181541 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.181508    5064 generic.go:356] "Generic (PLEG): container finished" podID="069d75fc-c250-45a6-8120-7157854e35d1" containerID="b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8" exitCode=0
Oct 30 15:29:28.181713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.181584    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerDied","Data":"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"}
Oct 30 15:29:28.182120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.182098    5064 scope.go:122] "RemoveContainer" containerID="b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"
Oct 30 15:29:28.184296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.184258    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:28.184397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.184354    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:28.184437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:28.184411    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:29:29.191664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:29.191625    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerStarted","Data":"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"}
Oct 30 15:29:29.660969 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:29.660926    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:29:30.189705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:30.189671    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:29:30.191159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:30.191127    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 15:29:30.191337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:30.191175    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 15:29:31.190320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:31.190277    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 15:29:31.190767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:31.190352    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 15:29:31.643172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:31.643132    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:29:32.190660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:32.190621    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 15:29:32.191092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:32.190704    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 15:29:32.763963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:32.763923    5064 scope.go:122] "RemoveContainer" containerID="7400023a7dd4894a5857a67bffe0c600d357702ef75fe92012256ed7ee563698"
Oct 30 15:29:33.189890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:33.189820    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 15:29:33.190079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:33.189912    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 15:29:34.190534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:34.190493    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 15:29:34.190945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:34.190550    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 15:29:35.246216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.246174    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:29:35.248280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.248250    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:29:35.728049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.728011    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:29:35.732778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.732749    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.750115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.750086    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-97ead4\"/\"default-dockercfg-7wc5k\""
Oct 30 15:29:35.750291 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.750226    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-97ead4\"/\"auth-test-model-97ead4-kserve-self-signed-certs\""
Oct 30 15:29:35.750373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.750293    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-97ead4\"/\"openshift-service-ca.crt\""
Oct 30 15:29:35.750435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.750375    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-97ead4\"/\"kube-root-ca.crt\""
Oct 30 15:29:35.755564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.754772    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:29:35.810480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.810441    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.810676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.810601    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.810816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.810790    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4mt\" (UniqueName: \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.811294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.810954    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.811294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.811178    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.811474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.811364    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.913491    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.913737    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.913812    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.913918    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4mt\" (UniqueName: \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.913979    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.914022    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.914626    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.914927    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.918339    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.918990    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.922850 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.920470    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:35.944142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:35.944104    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4mt\" (UniqueName: \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\") pod \"auth-test-model-97ead4-kserve-65659f68d5-8cn2c\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") " pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:36.053362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:36.053212    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:36.593387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:36.593353    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:29:36.594136 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:29:36.594109    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4022ae2e_c6a9_4e6e_97c4_4c6797e7974b.slice/crio-fc0a691f0e49366355c92dad2a1d0838e32bcd4dca53b005ac5fde62b5b8085b WatchSource:0}: Error finding container fc0a691f0e49366355c92dad2a1d0838e32bcd4dca53b005ac5fde62b5b8085b: Status 404 returned error can't find the container with id fc0a691f0e49366355c92dad2a1d0838e32bcd4dca53b005ac5fde62b5b8085b
Oct 30 15:29:37.310022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:37.309983    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" event={"ID":"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b","Type":"ContainerStarted","Data":"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"}
Oct 30 15:29:37.310022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:37.310029    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" event={"ID":"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b","Type":"ContainerStarted","Data":"fc0a691f0e49366355c92dad2a1d0838e32bcd4dca53b005ac5fde62b5b8085b"}
Oct 30 15:29:37.355973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:37.355923    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" podStartSLOduration=2.355906484 podStartE2EDuration="2.355906484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:37.353801901 +0000 UTC m=+2526.795692618" watchObservedRunningTime="2026-10-30 15:29:37.355906484 +0000 UTC m=+2526.797797222"
Oct 30 15:29:38.053918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:38.053878    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:46.053785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:46.053746    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:46.067406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:46.067370    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:46.402144 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:46.402112    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:29:48.332461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.332421    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:29:48.339005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.338975    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.365722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.365687    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:29:48.366247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.366221    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf437b-serving-cert\""
Oct 30 15:29:48.402565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.402533    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.402856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.402812    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.403138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.403111    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg26s\" (UniqueName: \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.403371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.403352    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.505411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.505364    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.505604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.505502    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.505604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.505551    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.505702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.505679    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg26s\" (UniqueName: \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.505800 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:48.505681    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-serving-cert: secret "maas-api-e2e-ipp-a-bf437b-serving-cert" not found
Oct 30 15:29:48.505918 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:48.505905    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls podName:57dd6fbb-e33d-445e-9374-0fc7f2a9939d nodeName:}" failed. No retries permitted until 2026-10-30 15:29:49.005880925 +0000 UTC m=+2538.447771625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls") pod "maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" (UID: "57dd6fbb-e33d-445e-9374-0fc7f2a9939d") : secret "maas-api-e2e-ipp-a-bf437b-serving-cert" not found
Oct 30 15:29:48.506468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.506439    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.508224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.508194    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:48.563725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:48.558252    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg26s\" (UniqueName: \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:49.015978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:49.015941    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:49.018710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:49.018680    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:49.260285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:49.260248    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:49.573365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:49.573335    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:29:49.574717 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:29:49.574688    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dd6fbb_e33d_445e_9374_0fc7f2a9939d.slice/crio-aeda86feae590219ee28fa657d715af05bc2442b0c1e4ce6818f7c120d0be7a3 WatchSource:0}: Error finding container aeda86feae590219ee28fa657d715af05bc2442b0c1e4ce6818f7c120d0be7a3: Status 404 returned error can't find the container with id aeda86feae590219ee28fa657d715af05bc2442b0c1e4ce6818f7c120d0be7a3
Oct 30 15:29:50.428697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:50.428649    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" event={"ID":"57dd6fbb-e33d-445e-9374-0fc7f2a9939d","Type":"ContainerStarted","Data":"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"}
Oct 30 15:29:50.428939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:50.428745    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:50.428939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:50.428765    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" event={"ID":"57dd6fbb-e33d-445e-9374-0fc7f2a9939d","Type":"ContainerStarted","Data":"aeda86feae590219ee28fa657d715af05bc2442b0c1e4ce6818f7c120d0be7a3"}
Oct 30 15:29:50.476296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:50.476232    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" podStartSLOduration=2.070282562 podStartE2EDuration="2.476216404s" totalImagesPullingTime="405.933842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:50.475201618 +0000 UTC m=+2539.917092337" watchObservedRunningTime="2026-10-30 15:29:50.476216404 +0000 UTC m=+2539.918107134"
Oct 30 15:29:56.442659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:29:56.442624    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:29:59.687661 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:29:59.687580    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:30:11.207867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.207767    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:30:11.222514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.222460    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.245629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.245596    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:30:11.280359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.280324    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a1654b-openshift-default-dockercfg-lpxrm\""
Oct 30 15:30:11.313172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313122    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313248    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313375    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313428    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313490    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313741    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.313910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.313795    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.314026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.314004    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frkd\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.314088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.314041    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418002    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418169    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418227    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frkd\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418274    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418358    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418411    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418462    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418497    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.418852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.418554    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.419756    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.420084    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.423455    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.424081    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.424338    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.432856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.428954    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.453851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.447619    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.482020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.479813    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frkd\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.482730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.482699    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.564862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.551191    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:11.928790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.928754    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:30:11.929551 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:30:11.929526    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a745e2d_c9e6_48fc_af52_8db0d737a05b.slice/crio-696f9cc9a531383daef2d251034798cfda4ff3f735c6fbfc77fa9493f2999dd1 WatchSource:0}: Error finding container 696f9cc9a531383daef2d251034798cfda4ff3f735c6fbfc77fa9493f2999dd1: Status 404 returned error can't find the container with id 696f9cc9a531383daef2d251034798cfda4ff3f735c6fbfc77fa9493f2999dd1
Oct 30 15:30:11.932488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.932455    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:30:11.932561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.932521    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:30:11.932561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:11.932555    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:30:12.794919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:12.794886    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerStarted","Data":"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"}
Oct 30 15:30:12.795450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:12.794948    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerStarted","Data":"696f9cc9a531383daef2d251034798cfda4ff3f735c6fbfc77fa9493f2999dd1"}
Oct 30 15:30:12.860488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:12.860432    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podStartSLOduration=1.860417946 podStartE2EDuration="1.860417946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:12.858725809 +0000 UTC m=+2562.300616553" watchObservedRunningTime="2026-10-30 15:30:12.860417946 +0000 UTC m=+2562.302308662"
Oct 30 15:30:13.552358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:13.552316    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:13.559638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:13.559606    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:13.803350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:13.803263    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:13.804334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:30:13.804311    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:30:29.631113 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:30:29.631062    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:30:59.632944 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:30:59.632896    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:31:26.044396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.044354    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:31:26.050523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.050492    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.075726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.075675    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:31:26.083626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.083595    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e5423a-serving-cert\""
Oct 30 15:31:26.127808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.127757    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.128728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.128123    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.128728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.128253    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm49\" (UniqueName: \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.128728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.128337    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.230243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.230202    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.230431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.230263    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.230431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.230336    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm49\" (UniqueName: \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.230431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.230395    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.231313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.231284    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.233067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.233044    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.233363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.233339    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.259454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.259421    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm49\" (UniqueName: \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\") pod \"maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.378615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.378549    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:26.744579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:26.744552    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:31:26.745892 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:31:26.745864    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510d7439_570f_47ac_b5eb_88c06d2e9e2b.slice/crio-96ea3c29f7264a4163829b019537efe46cc6aabf35cdc4ec8bde0bac807381b9 WatchSource:0}: Error finding container 96ea3c29f7264a4163829b019537efe46cc6aabf35cdc4ec8bde0bac807381b9: Status 404 returned error can't find the container with id 96ea3c29f7264a4163829b019537efe46cc6aabf35cdc4ec8bde0bac807381b9
Oct 30 15:31:27.684871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:27.684789    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" event={"ID":"510d7439-570f-47ac-b5eb-88c06d2e9e2b","Type":"ContainerStarted","Data":"948aa0e3dd339f4bd509eada47652baeebad9e05880195410479d4705ea037e3"}
Oct 30 15:31:27.685389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:27.684910    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" event={"ID":"510d7439-570f-47ac-b5eb-88c06d2e9e2b","Type":"ContainerStarted","Data":"96ea3c29f7264a4163829b019537efe46cc6aabf35cdc4ec8bde0bac807381b9"}
Oct 30 15:31:27.686065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:27.686037    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:27.737955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:27.737886    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" podStartSLOduration=1.374661664 podStartE2EDuration="1.737871181s" totalImagesPullingTime="363.209517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:27.733124419 +0000 UTC m=+2637.175015137" watchObservedRunningTime="2026-10-30 15:31:27.737871181 +0000 UTC m=+2637.179761911"
Oct 30 15:31:29.632650 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:31:29.632615    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:31:34.705338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:34.705310    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:31:35.076950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:35.076801    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:31:36.709947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.709900    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:31:36.930202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.928985    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.942362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.940021    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:31:36.951102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.950127    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-631d84-openshift-default-dockercfg-ns62r\""
Oct 30 15:31:36.997633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.997542    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.997633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.997593    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.997871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.997686    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.997871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.997796    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.998017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.997974    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.998162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.998140    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsxw\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.998231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.998202    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.998321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.998304    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:36.998391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:36.998370    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099384    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099439    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099546    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099576    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099716    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099764    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.099809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099790    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099922    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsxw\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099956    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.099957    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.100010    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.100037    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.100116    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.100643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.100620    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.102252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.102228    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.102420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.102398    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.124585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.124551    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.124985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.124957    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsxw\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.248735 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.248609    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:37.585159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.585124    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:31:37.585991 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:31:37.585961    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1021bdc0_8e0a_4881_93c6_c9d70bdb2c6f.slice/crio-ef02703c2ecc91e64ea8d35f64935659d6829e56dd027c6231b3a94fc844e3a0 WatchSource:0}: Error finding container ef02703c2ecc91e64ea8d35f64935659d6829e56dd027c6231b3a94fc844e3a0: Status 404 returned error can't find the container with id ef02703c2ecc91e64ea8d35f64935659d6829e56dd027c6231b3a94fc844e3a0
Oct 30 15:31:37.588876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.588818    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:31:37.588971 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.588920    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:31:37.588971 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.588962    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:31:37.884422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.884384    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerStarted","Data":"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"}
Oct 30 15:31:37.884949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.884437    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerStarted","Data":"ef02703c2ecc91e64ea8d35f64935659d6829e56dd027c6231b3a94fc844e3a0"}
Oct 30 15:31:37.937164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:37.937106    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podStartSLOduration=1.937089125 podStartE2EDuration="1.937089125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:37.934991554 +0000 UTC m=+2647.376882271" watchObservedRunningTime="2026-10-30 15:31:37.937089125 +0000 UTC m=+2647.378979841"
Oct 30 15:31:38.158915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:38.157190    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:31:38.250298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:38.249750    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:38.256383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:38.256352    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:38.893645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:38.893611    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:38.894976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:38.894950    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:31:50.099889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.099768    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" podUID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" containerName="main" containerID="cri-o://aca3758242b6fe3fa6921e2a64a1c6e4524c4583adf5b5a7135b808c3293bfce" gracePeriod=45
Oct 30 15:31:50.246419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.246377    5064 generic.go:356] "Generic (PLEG): container finished" podID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" containerID="aca3758242b6fe3fa6921e2a64a1c6e4524c4583adf5b5a7135b808c3293bfce" exitCode=0
Oct 30 15:31:50.246686 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.246439    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" event={"ID":"8abae952-eafb-4cc6-b1f9-edd5709d0ff2","Type":"ContainerDied","Data":"aca3758242b6fe3fa6921e2a64a1c6e4524c4583adf5b5a7135b808c3293bfce"}
Oct 30 15:31:50.423367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.423337    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:31:50.446928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.446890    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\" (UniqueName: \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447021    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447194    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447226    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447284    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447323    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\") pod \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\" (UID: \"8abae952-eafb-4cc6-b1f9-edd5709d0ff2\") "
Oct 30 15:31:50.447501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447470    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:50.447584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447501    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:50.447645 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.447596    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:50.448095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.448072    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:50.448216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.448111    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:50.448216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.448121    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:50.449518 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.449488    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:50.449625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.449511    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "kube-api-access-97pjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:31:50.449625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.449531    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs" pod "8abae952-eafb-4cc6-b1f9-edd5709d0ff2" (UID: "8abae952-eafb-4cc6-b1f9-edd5709d0ff2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:31:50.550490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.550448    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97pjl\" (UniqueName: \"kubernetes.io/projected/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-kube-api-access-97pjl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:50.550490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.550493    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:50.550695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:50.550510    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abae952-eafb-4cc6-b1f9-edd5709d0ff2-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:51.272357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:51.272321    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj" event={"ID":"8abae952-eafb-4cc6-b1f9-edd5709d0ff2","Type":"ContainerDied","Data":"333b1a966d06f6024532b88d7e505afa7c96e1f174366686e091365689fed379"}
Oct 30 15:31:51.273103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:51.273075    5064 scope.go:122] "RemoveContainer" containerID="aca3758242b6fe3fa6921e2a64a1c6e4524c4583adf5b5a7135b808c3293bfce"
Oct 30 15:31:51.273387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:51.272912    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"
Oct 30 15:31:51.336964 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:51.336923    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:31:51.350590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:51.350553    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-09e8eb/auth-test-model-09e8eb-kserve-f564fc869-5rncj"]
Oct 30 15:31:52.729324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:52.729278    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" path="/var/lib/kubelet/pods/8abae952-eafb-4cc6-b1f9-edd5709d0ff2/volumes"
Oct 30 15:31:53.201856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.201101    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerName="main" containerID="cri-o://418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5" gracePeriod=45
Oct 30 15:31:53.514579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.514550    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:31:53.617111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617072    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\" (UniqueName: \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617262    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617416    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617520    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617645    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617687    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\") pod \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\" (UID: \"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b\") "
Oct 30 15:31:53.617748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.617704    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:53.618075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.618045    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:53.618276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.618255    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:53.618603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.618580    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:53.618603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.618604    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:53.618742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.618616    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:53.620876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.620208    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:31:53.620876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.620216    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:31:53.620876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.620363    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt" pod "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" (UID: "4022ae2e-c6a9-4e6e-97c4-4c6797e7974b"). InnerVolumeSpecName "kube-api-access-mt4mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:31:53.719440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.719405    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:53.719440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.719435    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:53.719671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:53.719451    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt4mt\" (UniqueName: \"kubernetes.io/projected/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b-kube-api-access-mt4mt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:31:54.298560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.298519    5064 generic.go:356] "Generic (PLEG): container finished" podID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerID="418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5" exitCode=0
Oct 30 15:31:54.299075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.298568    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"
Oct 30 15:31:54.299075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.298602    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" event={"ID":"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b","Type":"ContainerDied","Data":"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"}
Oct 30 15:31:54.299075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.298644    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" event={"ID":"4022ae2e-c6a9-4e6e-97c4-4c6797e7974b","Type":"ContainerDied","Data":"fc0a691f0e49366355c92dad2a1d0838e32bcd4dca53b005ac5fde62b5b8085b"}
Oct 30 15:31:54.299075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.298663    5064 scope.go:122] "RemoveContainer" containerID="418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"
Oct 30 15:31:54.310475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.310449    5064 scope.go:122] "RemoveContainer" containerID="418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"
Oct 30 15:31:54.310855 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:31:54.310813    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5\": container with ID starting with 418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5 not found: ID does not exist" containerID="418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"
Oct 30 15:31:54.310985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.310865    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5"} err="failed to get container status \"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5\": rpc error: code = NotFound desc = could not find container \"418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5\": container with ID starting with 418659e7c9628b44867fc0de1793fcd9427f0a1bff72874a26639dd4016cd3e5 not found: ID does not exist"
Oct 30 15:31:54.361688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.361655    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:31:54.369314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.369288    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c"]
Oct 30 15:31:54.400615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.400574    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-97ead4/auth-test-model-97ead4-kserve-65659f68d5-8cn2c" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:31:54.724465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:31:54.724430    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" path="/var/lib/kubelet/pods/4022ae2e-c6a9-4e6e-97c4-4c6797e7974b/volumes"
Oct 30 15:31:59.759864 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:31:59.756311    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:32:10.432574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.432533    5064 generic.go:356] "Generic (PLEG): container finished" podID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerID="e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40" exitCode=0
Oct 30 15:32:10.433093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.432608    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerDied","Data":"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"}
Oct 30 15:32:10.433148 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.433125    5064 scope.go:122] "RemoveContainer" containerID="e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"
Oct 30 15:32:10.435209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.435177    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:32:10.435311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.435261    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:32:10.435311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:10.435293    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:32:11.495854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:11.490221    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerStarted","Data":"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"}
Oct 30 15:32:11.552540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:11.552503    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:32:11.553959 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:11.553930    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:11.554090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:11.553987    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:12.552720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:12.552686    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:12.553146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:12.552756    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:13.552401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:13.552356    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:13.552586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:13.552434    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:13.803853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:13.803762    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:32:14.552041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:14.552004    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:14.552241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:14.552063    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:15.552251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:15.552211    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:15.552777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:15.552271    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:16.552186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:16.552138    5064 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 15:32:16.552409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:16.552199    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 15:32:17.557702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:17.557670    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:32:17.558984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:17.558959    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:32:29.722166 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:32:29.722086    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:32:31.269539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:31.269507    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:32:31.271224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:31.271202    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:32:33.168783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.168743    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:32:33.169823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.169797    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" containerName="main"
Oct 30 15:32:33.169823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.169844    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" containerName="main"
Oct 30 15:32:33.170066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.169919    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerName="main"
Oct 30 15:32:33.170066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.169928    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerName="main"
Oct 30 15:32:33.170171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.170163    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abae952-eafb-4cc6-b1f9-edd5709d0ff2" containerName="main"
Oct 30 15:32:33.170224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.170189    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4022ae2e-c6a9-4e6e-97c4-4c6797e7974b" containerName="main"
Oct 30 15:32:33.175439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.175420    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.197921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.197889    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-631d84-serving-cert\""
Oct 30 15:32:33.198449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.198419    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:32:33.230596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.230563    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9m48\" (UniqueName: \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.230759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.230729    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.230916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.230897    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.230984 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.230957    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.332772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.332735    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9m48\" (UniqueName: \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.333019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.332861    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.333019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.332944    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.333019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.332993    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.333703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.333673    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.335483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.335458    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.335603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.335498    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.357804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.357768    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9m48\" (UniqueName: \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\") pod \"maas-api-e2e-ipp-b-631d84-558468cddd-78z5t\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.494916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.494297    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:33.833396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.833361    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:32:33.834607 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:32:33.834579    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728650fb_8a8b_439d_94d1_e8da3614699e.slice/crio-2cd91264e3d16225866c825819dccf4438a3cdfca6632a286c7309aa2bb7f4df WatchSource:0}: Error finding container 2cd91264e3d16225866c825819dccf4438a3cdfca6632a286c7309aa2bb7f4df: Status 404 returned error can't find the container with id 2cd91264e3d16225866c825819dccf4438a3cdfca6632a286c7309aa2bb7f4df
Oct 30 15:32:33.836215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:33.836197    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:32:34.769262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:34.769215    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" event={"ID":"728650fb-8a8b-439d-94d1-e8da3614699e","Type":"ContainerStarted","Data":"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"}
Oct 30 15:32:34.769664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:34.769288    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" event={"ID":"728650fb-8a8b-439d-94d1-e8da3614699e","Type":"ContainerStarted","Data":"2cd91264e3d16225866c825819dccf4438a3cdfca6632a286c7309aa2bb7f4df"}
Oct 30 15:32:34.769664 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:34.769322    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:34.818161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:34.818104    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" podStartSLOduration=1.414676349 podStartE2EDuration="1.818089336s" totalImagesPullingTime="403.412987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:34.81524282 +0000 UTC m=+2704.257133538" watchObservedRunningTime="2026-10-30 15:32:34.818089336 +0000 UTC m=+2704.259980053"
Oct 30 15:32:38.928090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:38.928048    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:32:38.933107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:38.933089    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:38.949416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:38.949384    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-78b513-serving-cert\""
Oct 30 15:32:38.955819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:38.955787    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:32:39.052030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.051993    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.052234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.052159    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.052303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.052237    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj755\" (UniqueName: \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.052370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.052335    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.154071 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.154030    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.154905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.154871    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj755\" (UniqueName: \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.155173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.155150    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.155555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.155532    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.157195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.156677    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.202843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.192304    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.204352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.197726    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj755\" (UniqueName: \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.204536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.203901    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.248950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.248917    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:39.567425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.567390    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:32:39.567565 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:32:39.567424    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a30de2c_790a_4a13_855f_5934160ea19c.slice/crio-3526f6e907f4b6b15787248e951bd268193d2b14cc798f334a12dc9d48f6dccc WatchSource:0}: Error finding container 3526f6e907f4b6b15787248e951bd268193d2b14cc798f334a12dc9d48f6dccc: Status 404 returned error can't find the container with id 3526f6e907f4b6b15787248e951bd268193d2b14cc798f334a12dc9d48f6dccc
Oct 30 15:32:39.827390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:39.827297    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" event={"ID":"4a30de2c-790a-4a13-855f-5934160ea19c","Type":"ContainerStarted","Data":"3526f6e907f4b6b15787248e951bd268193d2b14cc798f334a12dc9d48f6dccc"}
Oct 30 15:32:40.783800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:40.783767    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:32:40.837309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:40.837270    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" event={"ID":"4a30de2c-790a-4a13-855f-5934160ea19c","Type":"ContainerStarted","Data":"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"}
Oct 30 15:32:40.837487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:40.837367    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:40.887111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:40.887048    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" podStartSLOduration=2.5336544720000003 podStartE2EDuration="2.887029336s" totalImagesPullingTime="353.374864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:40.882788969 +0000 UTC m=+2710.324679690" watchObservedRunningTime="2026-10-30 15:32:40.887029336 +0000 UTC m=+2710.328920055"
Oct 30 15:32:46.865352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:32:46.862872    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:32:59.633337 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:32:59.633297    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:33:10.173145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.173106    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:33:10.179223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.179197    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.209562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.209524    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:33:10.312642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.312598    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.312642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.312646    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rlc\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.312913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.312886    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.312970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.312930    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.312970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.312958    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.313065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.313053    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.313105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.313085    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.313161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.313115    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.313352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.313330    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.414845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.414798    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.414877    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.414926    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415114    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415218    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415254    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rlc\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415258    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415407    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415437    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415460    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415468    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415591    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.415931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.415910    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.416181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.416160    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.417544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.417522    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.418016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.417998    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.456246 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.456118    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.494698 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.494664    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rlc\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.500807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.500773    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:10.835414 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.835382    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:33:10.835539 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:33:10.835407    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193959a9_6f5a_4d29_a1c0_be5433a7861f.slice/crio-5171e4220791df067547a876656bfcbb1d3a9c0e7f7dbcdce1faea20ee1d9ee7 WatchSource:0}: Error finding container 5171e4220791df067547a876656bfcbb1d3a9c0e7f7dbcdce1faea20ee1d9ee7: Status 404 returned error can't find the container with id 5171e4220791df067547a876656bfcbb1d3a9c0e7f7dbcdce1faea20ee1d9ee7
Oct 30 15:33:10.838433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.838389    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:10.838513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.838468    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:10.838513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:10.838498    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:11.268930 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.268878    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" event={"ID":"193959a9-6f5a-4d29-a1c0-be5433a7861f","Type":"ContainerStarted","Data":"37e063e38ccde8d356b01e40567faffabe7ce8304b5807a69b0f2981bddc164f"}
Oct 30 15:33:11.269389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.268952    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" event={"ID":"193959a9-6f5a-4d29-a1c0-be5433a7861f","Type":"ContainerStarted","Data":"5171e4220791df067547a876656bfcbb1d3a9c0e7f7dbcdce1faea20ee1d9ee7"}
Oct 30 15:33:11.318851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.318780    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podStartSLOduration=1.31876558 podStartE2EDuration="1.31876558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:11.316616128 +0000 UTC m=+2740.758506845" watchObservedRunningTime="2026-10-30 15:33:11.31876558 +0000 UTC m=+2740.760656298"
Oct 30 15:33:11.501568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.501531    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:11.502817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.502792    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:11.502938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:11.502856    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:12.502103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:12.502067    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:12.502516 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:12.502126    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:13.501973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:13.501931    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:13.502176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:13.501996    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:14.501292 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:14.501252    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:14.501511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:14.501314    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:15.501918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:15.501880    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:15.502373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:15.501943    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:16.501427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:16.501370    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 15:33:16.501653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:16.501467    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 15:33:17.506045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:17.506014    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:17.506548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:17.506528    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:17.507082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:17.507063    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:33:19.438710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.438669    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:33:19.444682 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.444653    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.526918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.520804    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b236a6-openshift-default-dockercfg-gnhzx\""
Oct 30 15:33:19.534010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.533968    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:33:19.539722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.539668    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.540166 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.540136    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.540500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.540479    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn862\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.540773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.540754    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.541898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.541871    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.542189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.542169    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.542891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.542821    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.543114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.543093    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.543556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.543528    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.644662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.644624    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.644883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.644671    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.644973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.644922    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.644973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.644963    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645026    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645115    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn862\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645208    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645237    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645352    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645390    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645456    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645759    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645778    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.645947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.645778    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.647293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.647264    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.647633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.647618    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.670772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.670735    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.670947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.670811    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn862\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:19.779626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:19.779522    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:20.111973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.111941    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:33:20.112258 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:33:20.112233    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c696c5_8483_48ba_8f51_afafcfb43d9f.slice/crio-0dba2cc267da1d7017e8dc165d0fd9fb59578c6e0523135366f5d814501a8f37 WatchSource:0}: Error finding container 0dba2cc267da1d7017e8dc165d0fd9fb59578c6e0523135366f5d814501a8f37: Status 404 returned error can't find the container with id 0dba2cc267da1d7017e8dc165d0fd9fb59578c6e0523135366f5d814501a8f37
Oct 30 15:33:20.115747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.115707    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:20.115826 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.115789    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:20.115826 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.115819    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:20.342555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.342520    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" event={"ID":"47c696c5-8483-48ba-8f51-afafcfb43d9f","Type":"ContainerStarted","Data":"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"}
Oct 30 15:33:20.342736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.342579    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" event={"ID":"47c696c5-8483-48ba-8f51-afafcfb43d9f","Type":"ContainerStarted","Data":"0dba2cc267da1d7017e8dc165d0fd9fb59578c6e0523135366f5d814501a8f37"}
Oct 30 15:33:20.393454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.393394    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podStartSLOduration=1.393379996 podStartE2EDuration="1.393379996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:20.390721266 +0000 UTC m=+2749.832611984" watchObservedRunningTime="2026-10-30 15:33:20.393379996 +0000 UTC m=+2749.835270712"
Oct 30 15:33:20.779975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:20.779886    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:21.782449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:21.782414    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:21.782899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:21.782468    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:22.780527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:22.780486    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:22.780731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:22.780547    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:23.780650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:23.780607    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:23.781156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:23.780687    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:24.779878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:24.779844    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:24.780079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:24.779908    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:25.780051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:25.780007    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:25.780475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:25.780077    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:26.780179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:26.780137    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 15:33:26.780581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:26.780202    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 15:33:27.812328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:27.812290    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:27.813716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:27.813693    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:27.815269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:27.815246    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:33:29.634183 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:33:29.634137    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:33:39.967318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:39.966938    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:33:39.967318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:39.967017    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:33:44.002085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.002052    5064 generic.go:356] "Generic (PLEG): container finished" podID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerID="94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477" exitCode=0
Oct 30 15:33:44.002579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.002130    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerDied","Data":"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"}
Oct 30 15:33:44.002721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.002704    5064 scope.go:122] "RemoveContainer" containerID="94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"
Oct 30 15:33:44.004877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.004813    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:44.004995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.004912    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:44.004995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:44.004954    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:33:45.020579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:45.020534    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerStarted","Data":"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"}
Oct 30 15:33:45.249764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:45.249727    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:33:45.249991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:45.249943    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:45.250049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:45.249983    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:46.249473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:46.249431    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:46.249935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:46.249499    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:47.249504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:47.249427    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:47.249504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:47.249498    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:48.249529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:48.249490    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:48.249734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:48.249558    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:49.250013 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:49.249968    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:49.250513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:49.250048    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:50.249073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:50.249030    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 15:33:50.249273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:50.249119    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 15:33:51.255488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:51.255455    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:33:51.256043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:51.255745    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:33:51.257412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:51.257384    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:33:52.535873 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:52.535797    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:33:52.536359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:52.536057    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" podUID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" containerName="maas-api" containerID="cri-o://a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114" gracePeriod=30
Oct 30 15:33:52.878757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:52.878726    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:33:53.024738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.024699    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\") pod \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") "
Oct 30 15:33:53.024973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.024862    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\") pod \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") "
Oct 30 15:33:53.025058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.024974    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\" (UniqueName: \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\") pod \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") "
Oct 30 15:33:53.025133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.025093    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\") pod \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\" (UID: \"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1\") "
Oct 30 15:33:53.025189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.025147    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca" pod "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" (UID: "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:33:53.025975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.025948    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:33:53.027237 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.027212    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls" pod "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" (UID: "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:33:53.027532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.027515    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs" pod "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" (UID: "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:33:53.027601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.027560    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs" pod "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" (UID: "4b1bf9bf-1396-4951-b1ee-e28b93e51ae1"). InnerVolumeSpecName "kube-api-access-5zrzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:33:53.130702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.130648    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrzs\" (UniqueName: \"kubernetes.io/projected/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-kube-api-access-5zrzs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:33:53.131040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.131016    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:33:53.131158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.131145    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:33:53.263856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.243377    5064 generic.go:356] "Generic (PLEG): container finished" podID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" containerID="a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114" exitCode=0
Oct 30 15:33:53.263856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.244924    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"
Oct 30 15:33:53.263856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.256888    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" event={"ID":"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1","Type":"ContainerDied","Data":"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"}
Oct 30 15:33:53.263856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.257025    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb" event={"ID":"4b1bf9bf-1396-4951-b1ee-e28b93e51ae1","Type":"ContainerDied","Data":"6903e3cf98b38cc79701d0284786f3a64a86a0659a02dd27942a2bc77a1bc696"}
Oct 30 15:33:53.263856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.257061    5064 scope.go:122] "RemoveContainer" containerID="a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"
Oct 30 15:33:53.410990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.410900    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:33:53.429762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.429254    5064 scope.go:122] "RemoveContainer" containerID="a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"
Oct 30 15:33:53.430655 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:33:53.430541    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114\": container with ID starting with a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114 not found: ID does not exist" containerID="a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"
Oct 30 15:33:53.430655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.430583    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114"} err="failed to get container status \"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114\": rpc error: code = NotFound desc = could not find container \"a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114\": container with ID starting with a43e3d9cb1699d3e97b6ebad423e4ef416dc22d3d88f6d52a64e69b721b87114 not found: ID does not exist"
Oct 30 15:33:53.432145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.432080    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aede6-66646f4599-qdpgb"]
Oct 30 15:33:53.905270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.904061    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:33:53.905804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.905344    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" containerName="maas-api"
Oct 30 15:33:53.905804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.905369    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" containerName="maas-api"
Oct 30 15:33:53.905804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.905587    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" containerName="maas-api"
Oct 30 15:33:53.910131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.910104    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:53.926927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.926801    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5e8bf0-serving-cert\""
Oct 30 15:33:53.949785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:53.949751    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:33:54.088871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.088811    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.089053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.088878    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.089053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.088985    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.089229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.089209    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvt4\" (UniqueName: \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.195943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.194590    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvt4\" (UniqueName: \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.195943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.194926    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.195943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.194966    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.195943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.195016    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.196285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.195962    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.196285 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:33:54.196225    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-serving-cert: secret "maas-api-e2e-ipp-a-5e8bf0-serving-cert" not found
Oct 30 15:33:54.196389 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:33:54.196295    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls podName:27da83b4-c7c3-4336-8a33-03504738c856 nodeName:}" failed. No retries permitted until 2026-10-30 15:33:54.696276115 +0000 UTC m=+2784.138166810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls") pod "maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" (UID: "27da83b4-c7c3-4336-8a33-03504738c856") : secret "maas-api-e2e-ipp-a-5e8bf0-serving-cert" not found
Oct 30 15:33:54.198742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.198719    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.220734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.220705    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvt4\" (UniqueName: \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.704622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.704584    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.707156 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.707131    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:54.725324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.725290    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1bf9bf-1396-4951-b1ee-e28b93e51ae1" path="/var/lib/kubelet/pods/4b1bf9bf-1396-4951-b1ee-e28b93e51ae1/volumes"
Oct 30 15:33:54.829818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:54.829780    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:55.146812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:55.146769    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:33:55.147605 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:33:55.147574    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27da83b4_c7c3_4336_8a33_03504738c856.slice/crio-ca975707ed5a30d3fda8c1b1ddaa4a7a6b3248c687a65bb382f49e21fe01e096 WatchSource:0}: Error finding container ca975707ed5a30d3fda8c1b1ddaa4a7a6b3248c687a65bb382f49e21fe01e096: Status 404 returned error can't find the container with id ca975707ed5a30d3fda8c1b1ddaa4a7a6b3248c687a65bb382f49e21fe01e096
Oct 30 15:33:55.295861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:55.289535    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" event={"ID":"27da83b4-c7c3-4336-8a33-03504738c856","Type":"ContainerStarted","Data":"ca975707ed5a30d3fda8c1b1ddaa4a7a6b3248c687a65bb382f49e21fe01e096"}
Oct 30 15:33:56.299330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:56.299294    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" event={"ID":"27da83b4-c7c3-4336-8a33-03504738c856","Type":"ContainerStarted","Data":"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"}
Oct 30 15:33:56.299794 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:56.299522    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:33:56.346704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:33:56.346609    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" podStartSLOduration=2.9827200510000003 podStartE2EDuration="3.34659188s" totalImagesPullingTime="363.871829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:56.345661198 +0000 UTC m=+2785.787551915" watchObservedRunningTime="2026-10-30 15:33:56.34659188 +0000 UTC m=+2785.788482598"
Oct 30 15:33:59.676539 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:33:59.674747    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:34:01.048853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:01.047413    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:34:01.048853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:01.047776    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" containerID="cri-o://7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700" gracePeriod=30
Oct 30 15:34:02.322505 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:02.322473    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:34:05.250536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:05.250500    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:34:05.250980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:05.250564    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:34:05.726320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:05.726279    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:34:05.726527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:05.726478    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy" containerID="cri-o://b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3" gracePeriod=30
Oct 30 15:34:06.350886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.350852    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:34:06.417362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.417320    5064 generic.go:356] "Generic (PLEG): container finished" podID="069d75fc-c250-45a6-8120-7157854e35d1" containerID="7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700" exitCode=0
Oct 30 15:34:06.417567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.417394    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"
Oct 30 15:34:06.417669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.417394    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerDied","Data":"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"}
Oct 30 15:34:06.417918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.417883    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58" event={"ID":"069d75fc-c250-45a6-8120-7157854e35d1","Type":"ContainerDied","Data":"0f0ff49b83acbd9012810df453adbd768f9c3abe97ddf93a2ee717f5aa516cc8"}
Oct 30 15:34:06.418048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.417939    5064 scope.go:122] "RemoveContainer" containerID="7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"
Oct 30 15:34:06.431594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.431568    5064 scope.go:122] "RemoveContainer" containerID="b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"
Oct 30 15:34:06.443958 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.443934    5064 scope.go:122] "RemoveContainer" containerID="7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"
Oct 30 15:34:06.444297 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:06.444275    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700\": container with ID starting with 7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700 not found: ID does not exist" containerID="7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"
Oct 30 15:34:06.444379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.444311    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700"} err="failed to get container status \"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700\": rpc error: code = NotFound desc = could not find container \"7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700\": container with ID starting with 7a0067a51e81257be02dccd75f2d2503407b046b99290ae57bd82b1786abf700 not found: ID does not exist"
Oct 30 15:34:06.444379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.444337    5064 scope.go:122] "RemoveContainer" containerID="b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"
Oct 30 15:34:06.444662 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:06.444632    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8\": container with ID starting with b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8 not found: ID does not exist" containerID="b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"
Oct 30 15:34:06.444729 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.444674    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8"} err="failed to get container status \"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8\": rpc error: code = NotFound desc = could not find container \"b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8\": container with ID starting with b2f4060ff4af9e7c8f896e6eefe0d8239dcd2f13a257c319a16ca59cf77438f8 not found: ID does not exist"
Oct 30 15:34:06.456706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.456680    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.456809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.456740    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.456809 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.456790    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.456985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.456965    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457068    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:06.457168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457119    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457130    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:06.457342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457171    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457195    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:06.457342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457221    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457268    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457455    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\") pod \"069d75fc-c250-45a6-8120-7157854e35d1\" (UID: \"069d75fc-c250-45a6-8120-7157854e35d1\") "
Oct 30 15:34:06.457626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.457539    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:34:06.458047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458015    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:06.458328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458291    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.458450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458332    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.458450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458349    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.458450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458363    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069d75fc-c250-45a6-8120-7157854e35d1-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.458450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.458377    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.460115 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.460086    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:06.460321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.460295    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:34:06.460590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.460568    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "kube-api-access-4785m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:06.461366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.461346    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token" pod "069d75fc-c250-45a6-8120-7157854e35d1" (UID: "069d75fc-c250-45a6-8120-7157854e35d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:06.560204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.560164    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069d75fc-c250-45a6-8120-7157854e35d1-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.560204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.560198    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4785m\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-kube-api-access-4785m\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.560204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.560209    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069d75fc-c250-45a6-8120-7157854e35d1-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.560449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.560217    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069d75fc-c250-45a6-8120-7157854e35d1-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:06.773116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.773076    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:34:06.787053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:06.787017    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7aede6-openshift-default-746b96b68c-zpn58"]
Oct 30 15:34:08.725744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:08.725710    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069d75fc-c250-45a6-8120-7157854e35d1" path="/var/lib/kubelet/pods/069d75fc-c250-45a6-8120-7157854e35d1/volumes"
Oct 30 15:34:11.131142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.131115    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:34:11.170209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170168    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170240    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170303    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170321    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170396    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170435    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170530    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:11.170612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170608    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170647    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:11.170789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170756    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.170962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170785    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:34:11.170962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170886    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:11.170962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170917    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:11.171130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.170984    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\") pod \"5aa26d82-61de-49b1-a280-cd57252bb6ad\" (UID: \"5aa26d82-61de-49b1-a280-cd57252bb6ad\") "
Oct 30 15:34:11.171948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.171913    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.171948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.171936    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.171948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.171949    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa26d82-61de-49b1-a280-cd57252bb6ad-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.172201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.171962    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.172201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.171973    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.172890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.172805    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:11.173098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.173081    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:34:11.173228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.173213    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:11.173346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.173326    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx" pod "5aa26d82-61de-49b1-a280-cd57252bb6ad" (UID: "5aa26d82-61de-49b1-a280-cd57252bb6ad"). InnerVolumeSpecName "kube-api-access-2bckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:11.273720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.273682    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.273720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.273717    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bckx\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-kube-api-access-2bckx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.273720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.273729    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.273997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.273742    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa26d82-61de-49b1-a280-cd57252bb6ad-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:11.507408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.507301    5064 generic.go:356] "Generic (PLEG): container finished" podID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerID="b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3" exitCode=0
Oct 30 15:34:11.507575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.507380    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" event={"ID":"5aa26d82-61de-49b1-a280-cd57252bb6ad","Type":"ContainerDied","Data":"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"}
Oct 30 15:34:11.507575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.507553    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv" event={"ID":"5aa26d82-61de-49b1-a280-cd57252bb6ad","Type":"ContainerDied","Data":"62f516c60e56cb2c3c6bab81edd24dd9fe3ce6eb74b64db8bd07f3a2a608ae07"}
Oct 30 15:34:11.507667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.507466    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"
Oct 30 15:34:11.507667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.507582    5064 scope.go:122] "RemoveContainer" containerID="b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"
Oct 30 15:34:11.520706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.520682    5064 scope.go:122] "RemoveContainer" containerID="b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"
Oct 30 15:34:11.521136 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:11.521114    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3\": container with ID starting with b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3 not found: ID does not exist" containerID="b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"
Oct 30 15:34:11.521182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.521147    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3"} err="failed to get container status \"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3\": rpc error: code = NotFound desc = could not find container \"b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3\": container with ID starting with b694c8674f3bb185c8bb701b8deb916c478926e900289ad46efe9b6950e255a3 not found: ID does not exist"
Oct 30 15:34:11.569019 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.568981    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:34:11.582233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:11.582193    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-rjkjv"]
Oct 30 15:34:12.751198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:12.749784    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" path="/var/lib/kubelet/pods/5aa26d82-61de-49b1-a280-cd57252bb6ad/volumes"
Oct 30 15:34:15.221014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:15.220975    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:34:15.541306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:15.541201    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy" containerID="cri-o://8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e" gracePeriod=30
Oct 30 15:34:20.476391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.476352    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:34:20.477230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.476566    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" podUID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" containerName="maas-api" containerID="cri-o://948aa0e3dd339f4bd509eada47652baeebad9e05880195410479d4705ea037e3" gracePeriod=30
Oct 30 15:34:20.600671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.600634    5064 generic.go:356] "Generic (PLEG): container finished" podID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" containerID="948aa0e3dd339f4bd509eada47652baeebad9e05880195410479d4705ea037e3" exitCode=0
Oct 30 15:34:20.600905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.600712    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" event={"ID":"510d7439-570f-47ac-b5eb-88c06d2e9e2b","Type":"ContainerDied","Data":"948aa0e3dd339f4bd509eada47652baeebad9e05880195410479d4705ea037e3"}
Oct 30 15:34:20.935425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.935396    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:34:20.939405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:20.939379    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:34:21.167877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.162695    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\") pod \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") "
Oct 30 15:34:21.295044 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.173846    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca" pod "510d7439-570f-47ac-b5eb-88c06d2e9e2b" (UID: "510d7439-570f-47ac-b5eb-88c06d2e9e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:34:21.295424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.295399    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.296169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.296145    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\") pod \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") "
Oct 30 15:34:21.297012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.296978    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.297551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.297530    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.297888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.297868    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\") pod \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") "
Oct 30 15:34:21.298094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.298071    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.298250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.298236    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.298489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.298453    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\" (UniqueName: \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\") pod \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\" (UID: \"510d7439-570f-47ac-b5eb-88c06d2e9e2b\") "
Oct 30 15:34:21.298740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.298721    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.299640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.299618    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.299980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.299851    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.300206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.300188    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\") pod \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\" (UID: \"8e449a94-3bb9-4085-b5b6-2517018d8ac1\") "
Oct 30 15:34:21.301605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.301572    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d7439-570f-47ac-b5eb-88c06d2e9e2b-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.302689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.296008    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:34:21.302812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.301898    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:21.309138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.309087    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:21.309373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.309350    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:21.316639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.311997    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:21.318017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.317988    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "kube-api-access-rhxc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:21.319669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.319200    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs" pod "510d7439-570f-47ac-b5eb-88c06d2e9e2b" (UID: "510d7439-570f-47ac-b5eb-88c06d2e9e2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:34:21.319669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.319231    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls" pod "510d7439-570f-47ac-b5eb-88c06d2e9e2b" (UID: "510d7439-570f-47ac-b5eb-88c06d2e9e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:34:21.319669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.319313    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:34:21.319669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.319320    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:21.320234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.320205    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49" pod "510d7439-570f-47ac-b5eb-88c06d2e9e2b" (UID: "510d7439-570f-47ac-b5eb-88c06d2e9e2b"). InnerVolumeSpecName "kube-api-access-fdm49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:34:21.321384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.321365    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo" pod "8e449a94-3bb9-4085-b5b6-2517018d8ac1" (UID: "8e449a94-3bb9-4085-b5b6-2517018d8ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:34:21.403710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403673    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403705    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403716    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403730    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403739    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhxc8\" (UniqueName: \"kubernetes.io/projected/8e449a94-3bb9-4085-b5b6-2517018d8ac1-kube-api-access-rhxc8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403749    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d7439-570f-47ac-b5eb-88c06d2e9e2b-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403759    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403766    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403776    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm49\" (UniqueName: \"kubernetes.io/projected/510d7439-570f-47ac-b5eb-88c06d2e9e2b-kube-api-access-fdm49\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403784    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403793    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.403968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.403805    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e449a94-3bb9-4085-b5b6-2517018d8ac1-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:34:21.633744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.631645    5064 generic.go:356] "Generic (PLEG): container finished" podID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerID="8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e" exitCode=0
Oct 30 15:34:21.638290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.634356    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" event={"ID":"8e449a94-3bb9-4085-b5b6-2517018d8ac1","Type":"ContainerDied","Data":"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"}
Oct 30 15:34:21.638290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.634490    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg" event={"ID":"8e449a94-3bb9-4085-b5b6-2517018d8ac1","Type":"ContainerDied","Data":"14fac4133f4f3fe5b6d2aeeea89f4e0b9a6fca361c260ec148443d6f7fb83b1a"}
Oct 30 15:34:21.638290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.634537    5064 scope.go:122] "RemoveContainer" containerID="8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"
Oct 30 15:34:21.638290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.634782    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"
Oct 30 15:34:21.664200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.650444    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9" event={"ID":"510d7439-570f-47ac-b5eb-88c06d2e9e2b","Type":"ContainerDied","Data":"96ea3c29f7264a4163829b019537efe46cc6aabf35cdc4ec8bde0bac807381b9"}
Oct 30 15:34:21.664200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.650296    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"
Oct 30 15:34:21.682455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.682425    5064 scope.go:122] "RemoveContainer" containerID="8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"
Oct 30 15:34:21.683870 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:21.682800    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e\": container with ID starting with 8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e not found: ID does not exist" containerID="8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"
Oct 30 15:34:21.683870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.682894    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e"} err="failed to get container status \"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e\": rpc error: code = NotFound desc = could not find container \"8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e\": container with ID starting with 8074d52e95d28b7296cc374dd90f38898d8f7cb3736d6b92387cb24a4944052e not found: ID does not exist"
Oct 30 15:34:21.683870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.682921    5064 scope.go:122] "RemoveContainer" containerID="948aa0e3dd339f4bd509eada47652baeebad9e05880195410479d4705ea037e3"
Oct 30 15:34:21.720897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.720825    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:34:21.724703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.724679    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5423a-84865bbcd6-5kxk9"]
Oct 30 15:34:21.760674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.760652    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:34:21.768316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:21.768294    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-8fttg"]
Oct 30 15:34:22.725214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:22.725179    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" path="/var/lib/kubelet/pods/510d7439-570f-47ac-b5eb-88c06d2e9e2b/volumes"
Oct 30 15:34:22.725780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:22.725763    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" path="/var/lib/kubelet/pods/8e449a94-3bb9-4085-b5b6-2517018d8ac1/volumes"
Oct 30 15:34:29.638780 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:29.638740    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.937879    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939208    5064 cpu_manager.go:414] "containerMap: removing container" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939238    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939281    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939290    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939356    5064 cpu_manager.go:414] "containerMap: removing container" podUID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" containerName="maas-api"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939364    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" containerName="maas-api"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939388    5064 cpu_manager.go:414] "containerMap: removing container" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939397    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939565    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939581    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e449a94-3bb9-4085-b5b6-2517018d8ac1" containerName="istio-proxy"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939598    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="510d7439-570f-47ac-b5eb-88c06d2e9e2b" containerName="maas-api"
Oct 30 15:34:55.939909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.939612    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa26d82-61de-49b1-a280-cd57252bb6ad" containerName="istio-proxy"
Oct 30 15:34:55.973195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.970172    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:55.994621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:55.994587    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-908638-openshift-default-dockercfg-g2fph\""
Oct 30 15:34:56.026606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.026570    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:34:56.223455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223358    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223402    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223540    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r2v\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223650    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223851 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223726    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223862    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.223995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.223982    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.224036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.224006    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.224089 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.224037    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.336206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.336167    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.336904    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.339112    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.339535    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r2v\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.339628    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.339708    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.339765    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.340020    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.340059    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.340480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.340107    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.341157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.341056    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.341157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.341090    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.341275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.341206    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.341535 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.341494    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.345384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.345303    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.352005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.351979    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.366287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.366256    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r2v\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.367203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.367180    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\") pod \"e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") " pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:56.659601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:56.659560    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:57.010768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:57.010729    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:34:57.011586 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:34:57.011560    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ca11fd_24f5_42f8_85ae_125c19ffaba5.slice/crio-452ebf2536ff0642e37896e6108a3e49699fdf5c83dc9dc784c44509d0bbec93 WatchSource:0}: Error finding container 452ebf2536ff0642e37896e6108a3e49699fdf5c83dc9dc784c44509d0bbec93: Status 404 returned error can't find the container with id 452ebf2536ff0642e37896e6108a3e49699fdf5c83dc9dc784c44509d0bbec93
Oct 30 15:34:57.014917 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:57.014872    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:34:57.015010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:57.014985    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:34:57.015051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:57.015033    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:34:57.054488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:57.054443    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerStarted","Data":"452ebf2536ff0642e37896e6108a3e49699fdf5c83dc9dc784c44509d0bbec93"}
Oct 30 15:34:58.075854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:58.071377    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerStarted","Data":"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"}
Oct 30 15:34:58.140474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:58.140421    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podStartSLOduration=3.140406531 podStartE2EDuration="3.140406531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:58.138605146 +0000 UTC m=+2847.580495876" watchObservedRunningTime="2026-10-30 15:34:58.140406531 +0000 UTC m=+2847.582297311"
Oct 30 15:34:58.660514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:58.660470    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:58.667791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:58.667758    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:59.083469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:59.083385    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:59.084383 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:34:59.084361    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:34:59.632960 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:34:59.632911    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:35:08.411605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:08.411568    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:35:08.412139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:08.411819    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" podUID="598bc32b-9092-41c0-a22d-826b7b937abc" containerName="maas-api" containerID="cri-o://75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0" gracePeriod=30
Oct 30 15:35:08.998052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:08.997861    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:35:09.038531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.038487    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\") pod \"598bc32b-9092-41c0-a22d-826b7b937abc\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") "
Oct 30 15:35:09.038869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.038841    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\" (UniqueName: \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\") pod \"598bc32b-9092-41c0-a22d-826b7b937abc\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") "
Oct 30 15:35:09.039036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.039020    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\") pod \"598bc32b-9092-41c0-a22d-826b7b937abc\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") "
Oct 30 15:35:09.055087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.039390    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\") pod \"598bc32b-9092-41c0-a22d-826b7b937abc\" (UID: \"598bc32b-9092-41c0-a22d-826b7b937abc\") "
Oct 30 15:35:09.056619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.056582    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs" pod "598bc32b-9092-41c0-a22d-826b7b937abc" (UID: "598bc32b-9092-41c0-a22d-826b7b937abc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:35:09.062319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.060340    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca" pod "598bc32b-9092-41c0-a22d-826b7b937abc" (UID: "598bc32b-9092-41c0-a22d-826b7b937abc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:35:09.068253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.067933    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls" pod "598bc32b-9092-41c0-a22d-826b7b937abc" (UID: "598bc32b-9092-41c0-a22d-826b7b937abc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:35:09.068577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.068541    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg" pod "598bc32b-9092-41c0-a22d-826b7b937abc" (UID: "598bc32b-9092-41c0-a22d-826b7b937abc"). InnerVolumeSpecName "kube-api-access-szrbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:35:09.165093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.159092    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:09.165093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.159135    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szrbg\" (UniqueName: \"kubernetes.io/projected/598bc32b-9092-41c0-a22d-826b7b937abc-kube-api-access-szrbg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:09.165093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.159150    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/598bc32b-9092-41c0-a22d-826b7b937abc-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:09.165093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.159163    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/598bc32b-9092-41c0-a22d-826b7b937abc-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:09.224418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.224382    5064 generic.go:356] "Generic (PLEG): container finished" podID="598bc32b-9092-41c0-a22d-826b7b937abc" containerID="75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0" exitCode=0
Oct 30 15:35:09.224657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.224427    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"
Oct 30 15:35:09.224657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.224471    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" event={"ID":"598bc32b-9092-41c0-a22d-826b7b937abc","Type":"ContainerDied","Data":"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"}
Oct 30 15:35:09.224657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.224531    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll" event={"ID":"598bc32b-9092-41c0-a22d-826b7b937abc","Type":"ContainerDied","Data":"091807d7b873fcef2ececeb96ba501993d3df0f2271ec50117ae1edcebed2e6e"}
Oct 30 15:35:09.224657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.224562    5064 scope.go:122] "RemoveContainer" containerID="75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"
Oct 30 15:35:09.236419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.236393    5064 scope.go:122] "RemoveContainer" containerID="75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"
Oct 30 15:35:09.236698 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:09.236677    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0\": container with ID starting with 75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0 not found: ID does not exist" containerID="75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"
Oct 30 15:35:09.236785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.236707    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0"} err="failed to get container status \"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0\": rpc error: code = NotFound desc = could not find container \"75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0\": container with ID starting with 75ed2bcc5281abb919de9a8c1f25d003701f17eb2eaf64660a6af8f2f77302b0 not found: ID does not exist"
Oct 30 15:35:09.285244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.285202    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:35:09.289276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:09.289244    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-416d92-79b89888d6-gfqll"]
Oct 30 15:35:10.727103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:10.727067    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598bc32b-9092-41c0-a22d-826b7b937abc" path="/var/lib/kubelet/pods/598bc32b-9092-41c0-a22d-826b7b937abc/volumes"
Oct 30 15:35:16.389843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:16.389789    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:35:16.390369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:16.390061    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" containerID="cri-o://2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323" gracePeriod=30
Oct 30 15:35:20.043485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:20.043446    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:35:20.043931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:20.043525    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:35:21.698373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.698344    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:35:21.790921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.790882    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.790938    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791037    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791084    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791133    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791202    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791261    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:35:21.791481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791451    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791492    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:35:21.791544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791515    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.791660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.791547    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\") pod \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\" (UID: \"46b564f0-6bd7-4ea3-b5e1-6c104a805240\") "
Oct 30 15:35:21.792434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.792390    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:35:21.792572 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.792507    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.792647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.792529    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.793382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.793352    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:35:21.793861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.793806    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:35:21.798251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.798207    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:35:21.798525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.798271    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:35:21.798525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.798436    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "kube-api-access-jg66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:35:21.800202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.800168    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy" pod "46b564f0-6bd7-4ea3-b5e1-6c104a805240" (UID: "46b564f0-6bd7-4ea3-b5e1-6c104a805240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:35:21.894197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894136    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894178    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894194    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894208    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg66j\" (UniqueName: \"kubernetes.io/projected/46b564f0-6bd7-4ea3-b5e1-6c104a805240-kube-api-access-jg66j\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894224    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894238    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b564f0-6bd7-4ea3-b5e1-6c104a805240-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:21.894550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:21.894252    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b564f0-6bd7-4ea3-b5e1-6c104a805240-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:35:22.329442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.329410    5064 generic.go:356] "Generic (PLEG): container finished" podID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerID="2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323" exitCode=0
Oct 30 15:35:22.329634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.329463    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"
Oct 30 15:35:22.329634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.329492    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerDied","Data":"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"}
Oct 30 15:35:22.329634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.329555    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj" event={"ID":"46b564f0-6bd7-4ea3-b5e1-6c104a805240","Type":"ContainerDied","Data":"e2b15527ebd8f0f3af5276cbcd8341bfe06f76d24ca89a2fb63e024ee5b993a6"}
Oct 30 15:35:22.329634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.329585    5064 scope.go:122] "RemoveContainer" containerID="2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"
Oct 30 15:35:22.342881 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.342821    5064 scope.go:122] "RemoveContainer" containerID="c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"
Oct 30 15:35:22.354947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.354922    5064 scope.go:122] "RemoveContainer" containerID="2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"
Oct 30 15:35:22.355296 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:22.355277    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323\": container with ID starting with 2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323 not found: ID does not exist" containerID="2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"
Oct 30 15:35:22.355344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.355308    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323"} err="failed to get container status \"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323\": rpc error: code = NotFound desc = could not find container \"2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323\": container with ID starting with 2807aa73c4420ea3d8bc9564f4855854b3b24a6d97a4fc639d36a93a82cf7323 not found: ID does not exist"
Oct 30 15:35:22.355344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.355327    5064 scope.go:122] "RemoveContainer" containerID="c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"
Oct 30 15:35:22.355599 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:22.355582    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9\": container with ID starting with c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9 not found: ID does not exist" containerID="c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"
Oct 30 15:35:22.355649 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.355603    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9"} err="failed to get container status \"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9\": rpc error: code = NotFound desc = could not find container \"c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9\": container with ID starting with c11346db5ff0eccc12c834450b13e222882f46dec62d7caa345fdcef1662b9a9 not found: ID does not exist"
Oct 30 15:35:22.390020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.389976    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:35:22.403521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.403489    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-5lxlj"]
Oct 30 15:35:22.725703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:22.725666    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" path="/var/lib/kubelet/pods/46b564f0-6bd7-4ea3-b5e1-6c104a805240/volumes"
Oct 30 15:35:29.635923 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:29.635886    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:35:43.608040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608005    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:35:43.608804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608789    5064 cpu_manager.go:414] "containerMap: removing container" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:43.608804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608805    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608817    5064 cpu_manager.go:414] "containerMap: removing container" podUID="598bc32b-9092-41c0-a22d-826b7b937abc" containerName="maas-api"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608823    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598bc32b-9092-41c0-a22d-826b7b937abc" containerName="maas-api"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608862    5064 cpu_manager.go:414] "containerMap: removing container" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608868    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608874    5064 cpu_manager.go:414] "containerMap: removing container" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:35:43.608927 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.608879    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:35:43.609126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.609015    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:43.609126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.609024    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="598bc32b-9092-41c0-a22d-826b7b937abc" containerName="maas-api"
Oct 30 15:35:43.609126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.609034    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="069d75fc-c250-45a6-8120-7157854e35d1" containerName="istio-proxy"
Oct 30 15:35:43.613801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.613785    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.632734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.632712    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0644a3-openshift-default-dockercfg-bkv67\""
Oct 30 15:35:43.647932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.647898    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648017    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648585    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648624    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648700    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648723 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648715    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648795    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648846    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.648902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.648882    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4zs\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.651185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.651161    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:35:43.788621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.781600    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.788983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.788942    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.792852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.789180    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4zs\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.811623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.811497    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.811947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.811859    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.812402 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.812288    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.813112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.813088    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.813283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.813266    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.829856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.814213    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.832164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.814939    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.832399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.814966    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.832541 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.815302    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.832681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.832662    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.849883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.834174    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.885646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.885611    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4zs\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.903267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.902543    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.907078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.907040    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.909009 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.908970    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\") pod \"e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") " pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:43.930751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:43.930370    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:44.315525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.315493    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:35:44.317179 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:35:44.317151    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227280a0_0c75_47d4_81c2_8c2807a4ea26.slice/crio-bd6dee10b32d719856c7d9d4b1455a1a48ffb0a6fe207974b5737f8545c8a7d3 WatchSource:0}: Error finding container bd6dee10b32d719856c7d9d4b1455a1a48ffb0a6fe207974b5737f8545c8a7d3: Status 404 returned error can't find the container with id bd6dee10b32d719856c7d9d4b1455a1a48ffb0a6fe207974b5737f8545c8a7d3
Oct 30 15:35:44.320644 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.320609    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:35:44.320728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.320680    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:35:44.320728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.320708    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:35:44.578359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.578258    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerStarted","Data":"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"}
Oct 30 15:35:44.578359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.578315    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerStarted","Data":"bd6dee10b32d719856c7d9d4b1455a1a48ffb0a6fe207974b5737f8545c8a7d3"}
Oct 30 15:35:44.640994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.640918    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podStartSLOduration=1.640894079 podStartE2EDuration="1.640894079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:44.637630422 +0000 UTC m=+2894.079521140" watchObservedRunningTime="2026-10-30 15:35:44.640894079 +0000 UTC m=+2894.082784800"
Oct 30 15:35:44.931366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:44.931328    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:45.937730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:45.937690    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:46.635595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:46.635556    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:46.636661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:46.636631    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:35:58.244419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.244381    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:35:58.245338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.245320    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b564f0-6bd7-4ea3-b5e1-6c104a805240" containerName="istio-proxy"
Oct 30 15:35:58.250353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.250331    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.272515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.272477    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:35:58.280740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.280693    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-908638-serving-cert\""
Oct 30 15:35:58.480327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.480202    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.480674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.480649    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.480823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.480808    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzwl\" (UniqueName: \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.480823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.480870    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.582910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.582789    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.583100 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:58.582935    5064 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-serving-cert: secret "maas-api-e2e-ipp-b-908638-serving-cert" not found
Oct 30 15:35:58.583100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.582940    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.583100 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:58.583016    5064 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls podName:92a3593c-2d8a-4b90-ad42-ee87b0274f71 nodeName:}" failed. No retries permitted until 2026-10-30 15:35:59.082988432 +0000 UTC m=+2908.524879144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls") pod "maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" (UID: "92a3593c-2d8a-4b90-ad42-ee87b0274f71") : secret "maas-api-e2e-ipp-b-908638-serving-cert" not found
Oct 30 15:35:58.583312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.583151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzwl\" (UniqueName: \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.583312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.583197    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.583697 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.583673    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.585910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.585884    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:58.611403 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:58.611364    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzwl\" (UniqueName: \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:59.091737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:59.091708    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:59.094382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:59.094358    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") pod \"maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:59.167812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:59.167767    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:35:59.540553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:59.540519    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:35:59.541455 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:35:59.541427    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a3593c_2d8a_4b90_ad42_ee87b0274f71.slice/crio-dccb8248eb11b5bf3791d5d6caae5416db069c0393160bee48d4fe1fe95e8630 WatchSource:0}: Error finding container dccb8248eb11b5bf3791d5d6caae5416db069c0393160bee48d4fe1fe95e8630: Status 404 returned error can't find the container with id dccb8248eb11b5bf3791d5d6caae5416db069c0393160bee48d4fe1fe95e8630
Oct 30 15:35:59.770542 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:35:59.770501    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:35:59.778640 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:35:59.778602    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" event={"ID":"92a3593c-2d8a-4b90-ad42-ee87b0274f71","Type":"ContainerStarted","Data":"dccb8248eb11b5bf3791d5d6caae5416db069c0393160bee48d4fe1fe95e8630"}
Oct 30 15:36:00.789993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:00.789946    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" event={"ID":"92a3593c-2d8a-4b90-ad42-ee87b0274f71","Type":"ContainerStarted","Data":"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"}
Oct 30 15:36:00.790503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:00.790161    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:36:00.838673 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:00.838622    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" podStartSLOduration=2.422478882 podStartE2EDuration="2.838606686s" totalImagesPullingTime="416.127804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:00.834665466 +0000 UTC m=+2910.276556222" watchObservedRunningTime="2026-10-30 15:36:00.838606686 +0000 UTC m=+2910.280497402"
Oct 30 15:36:06.805943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:06.805909    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:36:12.491598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.491508    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:36:12.496600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.496572    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511207    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511296    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511390    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511506    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nqg\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511588    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511679    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511889    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.511950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.511936    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.512035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.512021    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.523985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.523958    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-261ad0-openshift-default-dockercfg-htgfx\""
Oct 30 15:36:12.542452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.542424    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:36:12.623254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.623213    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.807172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.807072    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.818948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.817325    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.819195 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.818927    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.820137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.820110    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.830058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.830019    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.833970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.833938    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.858294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.858257    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.858622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.858596    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nqg\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.858803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.858785    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.871855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.695586    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.917948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.833670    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.917948 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.859535    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.918228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.819253    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:12.918228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:12.860099    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:13.006932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.001664    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nqg\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:13.262429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.258852    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:13.280314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.278018    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\") pod \"e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") " pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:13.411861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.411803    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:13.739662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.739634    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:36:13.740891 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:36:13.740854    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7176ec41_5592_4f76_b284_203843247cf8.slice/crio-98963b8a54ae986488e3a7331943e6521d5c0eff81dd83c47374c0d99ac559e2 WatchSource:0}: Error finding container 98963b8a54ae986488e3a7331943e6521d5c0eff81dd83c47374c0d99ac559e2: Status 404 returned error can't find the container with id 98963b8a54ae986488e3a7331943e6521d5c0eff81dd83c47374c0d99ac559e2
Oct 30 15:36:13.744443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.744408    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:36:13.744547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.744482    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:36:13.744547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:13.744509    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:36:14.178659 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:14.178622    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerStarted","Data":"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"}
Oct 30 15:36:14.178880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:14.178681    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerStarted","Data":"98963b8a54ae986488e3a7331943e6521d5c0eff81dd83c47374c0d99ac559e2"}
Oct 30 15:36:14.230158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:14.230094    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podStartSLOduration=2.230075434 podStartE2EDuration="2.230075434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:14.226353992 +0000 UTC m=+2923.668244709" watchObservedRunningTime="2026-10-30 15:36:14.230075434 +0000 UTC m=+2923.671966150"
Oct 30 15:36:14.437408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:14.425490    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:14.443745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:14.443710    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:15.132985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.132947    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:36:15.133581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.133218    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" podUID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" containerName="maas-api" containerID="cri-o://d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8" gracePeriod=30
Oct 30 15:36:15.189347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.189310    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:15.190404 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.190376    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:36:15.463977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.463951    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:36:15.546336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.546301    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\" (UniqueName: \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\") pod \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") "
Oct 30 15:36:15.546536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.546396    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\") pod \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") "
Oct 30 15:36:15.546536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.546495    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") pod \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") "
Oct 30 15:36:15.546696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.546669    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\") pod \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\" (UID: \"57dd6fbb-e33d-445e-9374-0fc7f2a9939d\") "
Oct 30 15:36:15.547142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.547111    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca" pod "57dd6fbb-e33d-445e-9374-0fc7f2a9939d" (UID: "57dd6fbb-e33d-445e-9374-0fc7f2a9939d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:36:15.547601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.547580    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:15.548641 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.548616    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s" pod "57dd6fbb-e33d-445e-9374-0fc7f2a9939d" (UID: "57dd6fbb-e33d-445e-9374-0fc7f2a9939d"). InnerVolumeSpecName "kube-api-access-pg26s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:36:15.549091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.549067    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs" pod "57dd6fbb-e33d-445e-9374-0fc7f2a9939d" (UID: "57dd6fbb-e33d-445e-9374-0fc7f2a9939d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:36:15.549182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.549098    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls" pod "57dd6fbb-e33d-445e-9374-0fc7f2a9939d" (UID: "57dd6fbb-e33d-445e-9374-0fc7f2a9939d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:36:15.649585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.649539    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg26s\" (UniqueName: \"kubernetes.io/projected/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-kube-api-access-pg26s\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:15.649585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.649585    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:15.671100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:15.649602    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57dd6fbb-e33d-445e-9374-0fc7f2a9939d-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:16.199049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.199013    5064 generic.go:356] "Generic (PLEG): container finished" podID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" containerID="d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8" exitCode=0
Oct 30 15:36:16.199564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.199066    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"
Oct 30 15:36:16.199564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.199101    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" event={"ID":"57dd6fbb-e33d-445e-9374-0fc7f2a9939d","Type":"ContainerDied","Data":"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"}
Oct 30 15:36:16.199564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.199169    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht" event={"ID":"57dd6fbb-e33d-445e-9374-0fc7f2a9939d","Type":"ContainerDied","Data":"aeda86feae590219ee28fa657d715af05bc2442b0c1e4ce6818f7c120d0be7a3"}
Oct 30 15:36:16.199564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.199204    5064 scope.go:122] "RemoveContainer" containerID="d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"
Oct 30 15:36:16.211674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.211653    5064 scope.go:122] "RemoveContainer" containerID="d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"
Oct 30 15:36:16.212000 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:16.211977    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8\": container with ID starting with d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8 not found: ID does not exist" containerID="d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"
Oct 30 15:36:16.212090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.212007    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8"} err="failed to get container status \"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8\": rpc error: code = NotFound desc = could not find container \"d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8\": container with ID starting with d0e41d27907ca5961a21e9c1c91f1b2196ec87f5d999ef22041e9b26f5120bc8 not found: ID does not exist"
Oct 30 15:36:16.273934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.273901    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:36:16.278171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.278141    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf437b-7444886cd9-bcnht"]
Oct 30 15:36:16.725822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:16.725784    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" path="/var/lib/kubelet/pods/57dd6fbb-e33d-445e-9374-0fc7f2a9939d/volumes"
Oct 30 15:36:17.955637 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:17.955607    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:36:18.160033 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:18.160005    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:36:25.609763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:25.609719    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:36:25.610298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:25.610161    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" podUID="c94160f9-510f-4ff4-a8b5-a8be745da01d" containerName="maas-api" containerID="cri-o://da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a" gracePeriod=30
Oct 30 15:36:26.153330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.153293    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:36:26.242853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.242795    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\") pod \"c94160f9-510f-4ff4-a8b5-a8be745da01d\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") "
Oct 30 15:36:26.243239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.243213    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\") pod \"c94160f9-510f-4ff4-a8b5-a8be745da01d\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") "
Oct 30 15:36:26.243892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.243869    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") pod \"c94160f9-510f-4ff4-a8b5-a8be745da01d\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") "
Oct 30 15:36:26.267585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.244062    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\" (UniqueName: \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\") pod \"c94160f9-510f-4ff4-a8b5-a8be745da01d\" (UID: \"c94160f9-510f-4ff4-a8b5-a8be745da01d\") "
Oct 30 15:36:26.267956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.244671    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca" pod "c94160f9-510f-4ff4-a8b5-a8be745da01d" (UID: "c94160f9-510f-4ff4-a8b5-a8be745da01d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:36:26.268250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.268203    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs" pod "c94160f9-510f-4ff4-a8b5-a8be745da01d" (UID: "c94160f9-510f-4ff4-a8b5-a8be745da01d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:36:26.269021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.268992    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:26.269128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.269027    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94160f9-510f-4ff4-a8b5-a8be745da01d-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:26.331866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.324048    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls" pod "c94160f9-510f-4ff4-a8b5-a8be745da01d" (UID: "c94160f9-510f-4ff4-a8b5-a8be745da01d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:36:26.331866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.325709    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf" pod "c94160f9-510f-4ff4-a8b5-a8be745da01d" (UID: "c94160f9-510f-4ff4-a8b5-a8be745da01d"). InnerVolumeSpecName "kube-api-access-lh5gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:36:26.365696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.365643    5064 generic.go:356] "Generic (PLEG): container finished" podID="c94160f9-510f-4ff4-a8b5-a8be745da01d" containerID="da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a" exitCode=0
Oct 30 15:36:26.370074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.370022    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" event={"ID":"c94160f9-510f-4ff4-a8b5-a8be745da01d","Type":"ContainerDied","Data":"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"}
Oct 30 15:36:26.370381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.370356    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db" event={"ID":"c94160f9-510f-4ff4-a8b5-a8be745da01d","Type":"ContainerDied","Data":"ed6649d0e6f568c89cccec8a6747a0fe02da87c4e0adc8cd4817a3d7c14b14c9"}
Oct 30 15:36:26.370517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.370501    5064 scope.go:122] "RemoveContainer" containerID="da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"
Oct 30 15:36:26.370876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.370853    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"
Oct 30 15:36:26.376214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.375909    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94160f9-510f-4ff4-a8b5-a8be745da01d-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:26.376214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.375943    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5gf\" (UniqueName: \"kubernetes.io/projected/c94160f9-510f-4ff4-a8b5-a8be745da01d-kube-api-access-lh5gf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:26.396023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.395989    5064 scope.go:122] "RemoveContainer" containerID="da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"
Oct 30 15:36:26.396459 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:26.396435    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a\": container with ID starting with da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a not found: ID does not exist" containerID="da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"
Oct 30 15:36:26.396559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.396472    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a"} err="failed to get container status \"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a\": rpc error: code = NotFound desc = could not find container \"da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a\": container with ID starting with da831c5b7d31d72cf5bb045713e8c3264e8b0b352cb04f906829b9f2bd6be49a not found: ID does not exist"
Oct 30 15:36:26.439752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.439708    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:36:26.447679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.447650    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-100b2e-7b5ff957d7-dg5db"]
Oct 30 15:36:26.729770 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:26.729733    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94160f9-510f-4ff4-a8b5-a8be745da01d" path="/var/lib/kubelet/pods/c94160f9-510f-4ff4-a8b5-a8be745da01d/volumes"
Oct 30 15:36:29.632239 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:29.632199    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:36:34.182869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:34.182812    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:36:34.183799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:34.183738    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy" containerID="cri-o://81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509" gracePeriod=30
Oct 30 15:36:39.427457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.427427    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487320    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487394    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487451    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487566    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487642    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487687    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.487903    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.488070    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.488924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.488126    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\") pod \"852e8709-0822-4ce0-be18-a8ac4c8885f0\" (UID: \"852e8709-0822-4ce0-be18-a8ac4c8885f0\") "
Oct 30 15:36:39.511189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.510737    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:36:39.512546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.512508    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:36:39.513002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.512977    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:36:39.515363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.515338    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:36:39.515739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.515721    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:36:39.521225 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.521161    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:36:39.534390 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.534351    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "kube-api-access-ljkg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:36:39.535214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.535184    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:36:39.549335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.549278    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy" pod "852e8709-0822-4ce0-be18-a8ac4c8885f0" (UID: "852e8709-0822-4ce0-be18-a8ac4c8885f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:36:39.585251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.585214    5064 generic.go:356] "Generic (PLEG): container finished" podID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerID="81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509" exitCode=0
Oct 30 15:36:39.585445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.585271    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"
Oct 30 15:36:39.585445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.585298    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerDied","Data":"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"}
Oct 30 15:36:39.585445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.585359    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd" event={"ID":"852e8709-0822-4ce0-be18-a8ac4c8885f0","Type":"ContainerDied","Data":"41fd83bda2118374128a98cccc10774525ef82ce5fe25e9497e5cc8c7e84c82d"}
Oct 30 15:36:39.585445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.585388    5064 scope.go:122] "RemoveContainer" containerID="81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"
Oct 30 15:36:39.590772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590744    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590772    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkg5\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-kube-api-access-ljkg5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590782    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590790    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590799    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590807    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852e8709-0822-4ce0-be18-a8ac4c8885f0-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590816    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852e8709-0822-4ce0-be18-a8ac4c8885f0-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590843    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.590987 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.590856    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852e8709-0822-4ce0-be18-a8ac4c8885f0-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:36:39.600022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.599998    5064 scope.go:122] "RemoveContainer" containerID="2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"
Oct 30 15:36:39.613211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.613178    5064 scope.go:122] "RemoveContainer" containerID="81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"
Oct 30 15:36:39.613559 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:39.613524    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509\": container with ID starting with 81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509 not found: ID does not exist" containerID="81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"
Oct 30 15:36:39.613724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.613697    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509"} err="failed to get container status \"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509\": rpc error: code = NotFound desc = could not find container \"81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509\": container with ID starting with 81447d0d21c6e8009290066f37413f3f69fbffd2d2a8381b03dd78aa09e10509 not found: ID does not exist"
Oct 30 15:36:39.613843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.613814    5064 scope.go:122] "RemoveContainer" containerID="2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"
Oct 30 15:36:39.614296 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:39.614269    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148\": container with ID starting with 2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148 not found: ID does not exist" containerID="2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"
Oct 30 15:36:39.614392 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.614305    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148"} err="failed to get container status \"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148\": rpc error: code = NotFound desc = could not find container \"2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148\": container with ID starting with 2ae6b6efc515789217b91d744e5ab017ebfc35f64f6c47404f767cb8582ab148 not found: ID does not exist"
Oct 30 15:36:39.649342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.649303    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:36:39.660339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:39.660304    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-4qbcd"]
Oct 30 15:36:40.725391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:40.725359    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" path="/var/lib/kubelet/pods/852e8709-0822-4ce0-be18-a8ac4c8885f0/volumes"
Oct 30 15:36:45.226591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.226548    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:36:45.227375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227358    5064 cpu_manager.go:414] "containerMap: removing container" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:36:45.227375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227375    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227390    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c94160f9-510f-4ff4-a8b5-a8be745da01d" containerName="maas-api"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227396    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94160f9-510f-4ff4-a8b5-a8be745da01d" containerName="maas-api"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227427    5064 cpu_manager.go:414] "containerMap: removing container" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227432    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227452    5064 cpu_manager.go:414] "containerMap: removing container" podUID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" containerName="maas-api"
Oct 30 15:36:45.227507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227459    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" containerName="maas-api"
Oct 30 15:36:45.227799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227617    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:36:45.227799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227633    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94160f9-510f-4ff4-a8b5-a8be745da01d" containerName="maas-api"
Oct 30 15:36:45.227799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.227656    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dd6fbb-e33d-445e-9374-0fc7f2a9939d" containerName="maas-api"
Oct 30 15:36:45.233385 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.233362    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.250380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.250350    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:36:45.258314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.258276    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0644a3-serving-cert\""
Oct 30 15:36:45.389484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.389441    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nvt\" (UniqueName: \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.389692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.389564    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.389692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.389669    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.389787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.389693    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.494159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.494052    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nvt\" (UniqueName: \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.499218 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.499178    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.500159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.500129    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.501238 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.501213    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.505133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.501080    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.547855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.542294    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.553990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.553946    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nvt\" (UniqueName: \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.558432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.558402    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:45.721821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.721779    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 15:36:45.850277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:45.850176    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:46.173147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:46.173117    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:36:46.174715 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:36:46.174688    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55005f68_0a05_4bc9_aabf_fcd422046bd6.slice/crio-c182c9e164b5797e7cd891145904045dad7e05e033c8bfd07c4118213a6e58a8 WatchSource:0}: Error finding container c182c9e164b5797e7cd891145904045dad7e05e033c8bfd07c4118213a6e58a8: Status 404 returned error can't find the container with id c182c9e164b5797e7cd891145904045dad7e05e033c8bfd07c4118213a6e58a8
Oct 30 15:36:46.645181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:46.645141    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" event={"ID":"55005f68-0a05-4bc9-aabf-fcd422046bd6","Type":"ContainerStarted","Data":"c182c9e164b5797e7cd891145904045dad7e05e033c8bfd07c4118213a6e58a8"}
Oct 30 15:36:47.654602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:47.654559    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" event={"ID":"55005f68-0a05-4bc9-aabf-fcd422046bd6","Type":"ContainerStarted","Data":"fafd1a50ea5232f65f16739a0921343ea3c923981edc864cf437c44d2b55d517"}
Oct 30 15:36:47.655070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:47.654630    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:47.704270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:47.704211    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" podStartSLOduration=2.279884011 podStartE2EDuration="2.704194556s" totalImagesPullingTime="424.310545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:47.700440844 +0000 UTC m=+2957.142331561" watchObservedRunningTime="2026-10-30 15:36:47.704194556 +0000 UTC m=+2957.146085272"
Oct 30 15:36:50.723099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:50.723035    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing" containerID="cri-o://c6fee53f30bb72e2a29ee8b754841b08e93ae5dee1acb8785be3a27b0cb5ec47" gracePeriod=25
Oct 30 15:36:53.343561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:53.343522    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:36:53.668331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:36:53.668300    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:36:59.658482 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:36:59.658439    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:37:03.343610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:03.343566    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:37:05.324731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.324692    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:37:05.325765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.325742    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="852e8709-0822-4ce0-be18-a8ac4c8885f0" containerName="istio-proxy"
Oct 30 15:37:05.330041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.330020    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.348558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.348525    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:37:05.368790 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.368753    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e30f62-serving-cert\""
Oct 30 15:37:05.431437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.431394    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzsld\" (UniqueName: \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.431653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.431588    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.431739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.431674    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.431739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.431715    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.533027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.532990    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.533283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.533063    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.533283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.533085    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.533428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.533366    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzsld\" (UniqueName: \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.533973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.533937    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.536369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.536342    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.539524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.539464    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.558391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.558356    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzsld\" (UniqueName: \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\") pod \"maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.649307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.649201    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:05.992430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:05.992395    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:37:05.993747 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:37:05.993714    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a78a95e_8f90_4712_ab7a_f93ee42fd54f.slice/crio-af6b168d6327953c8a0a2554b50981a79b742947914804fd4b3c079662ec5fbf WatchSource:0}: Error finding container af6b168d6327953c8a0a2554b50981a79b742947914804fd4b3c079662ec5fbf: Status 404 returned error can't find the container with id af6b168d6327953c8a0a2554b50981a79b742947914804fd4b3c079662ec5fbf
Oct 30 15:37:06.949691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:06.949649    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" event={"ID":"1a78a95e-8f90-4712-ab7a-f93ee42fd54f","Type":"ContainerStarted","Data":"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"}
Oct 30 15:37:06.950199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:06.949707    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" event={"ID":"1a78a95e-8f90-4712-ab7a-f93ee42fd54f","Type":"ContainerStarted","Data":"af6b168d6327953c8a0a2554b50981a79b742947914804fd4b3c079662ec5fbf"}
Oct 30 15:37:06.950199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:06.949775    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:07.003819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:07.003756    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" podStartSLOduration=1.684370965 podStartE2EDuration="2.003741381s" totalImagesPullingTime="319.370416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:06.9992598 +0000 UTC m=+2976.441150516" watchObservedRunningTime="2026-10-30 15:37:07.003741381 +0000 UTC m=+2976.445632097"
Oct 30 15:37:09.977755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.977720    5064 generic.go:356] "Generic (PLEG): container finished" podID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerID="99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914" exitCode=0
Oct 30 15:37:09.978262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.977775    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerDied","Data":"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"}
Oct 30 15:37:09.978328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.978310    5064 scope.go:122] "RemoveContainer" containerID="99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"
Oct 30 15:37:09.980437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.980403    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:09.980520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.980483    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:09.980520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:09.980516    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:10.992011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:10.991908    5064 generic.go:356] "Generic (PLEG): container finished" podID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerID="c6fee53f30bb72e2a29ee8b754841b08e93ae5dee1acb8785be3a27b0cb5ec47" exitCode=0
Oct 30 15:37:10.992508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:10.992081    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" event={"ID":"631c3edf-e93b-484d-ac7a-6dc6f91d8505","Type":"ContainerDied","Data":"c6fee53f30bb72e2a29ee8b754841b08e93ae5dee1acb8785be3a27b0cb5ec47"}
Oct 30 15:37:10.994273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:10.994247    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerStarted","Data":"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"}
Oct 30 15:37:11.208290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.208257    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 15:37:11.344911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.344471    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\" (UniqueName: \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\") pod \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") "
Oct 30 15:37:11.344911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.344564    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\") pod \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\" (UID: \"631c3edf-e93b-484d-ac7a-6dc6f91d8505\") "
Oct 30 15:37:11.345981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.345502    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume" pod "631c3edf-e93b-484d-ac7a-6dc6f91d8505" (UID: "631c3edf-e93b-484d-ac7a-6dc6f91d8505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:37:11.346130 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.345979    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631c3edf-e93b-484d-ac7a-6dc6f91d8505-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:37:11.348377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.348347    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47" pod "631c3edf-e93b-484d-ac7a-6dc6f91d8505" (UID: "631c3edf-e93b-484d-ac7a-6dc6f91d8505"). InnerVolumeSpecName "kube-api-access-k5r47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:37:11.447651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.447613    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5r47\" (UniqueName: \"kubernetes.io/projected/631c3edf-e93b-484d-ac7a-6dc6f91d8505-kube-api-access-k5r47\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:37:11.660214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.660167    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:37:11.661563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.661532    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:11.661712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:11.661582    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:12.003590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.003502    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"
Oct 30 15:37:12.004034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.003500    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt" event={"ID":"631c3edf-e93b-484d-ac7a-6dc6f91d8505","Type":"ContainerDied","Data":"b6352ccefea18d711350b9d58eee36274359a4f3bb1583b89b7057496d96c684"}
Oct 30 15:37:12.004034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.003647    5064 scope.go:122] "RemoveContainer" containerID="c6fee53f30bb72e2a29ee8b754841b08e93ae5dee1acb8785be3a27b0cb5ec47"
Oct 30 15:37:12.081147 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.081112    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 15:37:12.132418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.132381    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-4f5c30-6759c95f55-tqhxt"]
Oct 30 15:37:12.660362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.660319    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:12.660570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.660385    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:12.748601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.748569    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" path="/var/lib/kubelet/pods/631c3edf-e93b-484d-ac7a-6dc6f91d8505/volumes"
Oct 30 15:37:12.981445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:12.981360    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:37:13.660157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:13.660109    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:13.660597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:13.660178    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:14.083713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:14.083629    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:37:14.662233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:14.662191    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:14.662751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:14.662259    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:15.660465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:15.660424    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:15.660674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:15.660488    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:16.815854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:16.809954    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 15:37:16.815854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:16.810042    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 15:37:17.665865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:17.665816    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:37:17.667005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:17.666981    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:37:27.234490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.234452    5064 generic.go:356] "Generic (PLEG): container finished" podID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerID="5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54" exitCode=0
Oct 30 15:37:27.234993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.234526    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerDied","Data":"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"}
Oct 30 15:37:27.235055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.235040    5064 scope.go:122] "RemoveContainer" containerID="5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"
Oct 30 15:37:27.237133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.237098    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:27.237264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.237211    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:27.237331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:27.237265    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:37:28.244438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:28.244404    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerStarted","Data":"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"}
Oct 30 15:37:28.938743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:28.938366    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:37:28.940161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:28.940128    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:28.940314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:28.940179    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:29.660743 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:37:29.660698    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:37:29.931590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:29.931488    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:29.938030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:29.937988    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:30.932568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:30.932525    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:30.933294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:30.933243    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:31.354545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:31.354461    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:37:31.360405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:31.360375    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:37:31.637997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:31.637951    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:37:31.938466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:31.938367    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:31.938466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:31.938453    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:32.936962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:32.936915    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:32.937485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:32.937448    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:33.931624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:33.931581    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 15:37:33.932072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:33.931659    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 15:37:35.161578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:35.161542    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:37:35.164594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:37:35.164565    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:37:59.632760 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:37:59.632674    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:38:02.086778 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.086635    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:38:02.087694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.087335    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" podUID="3c51e121-fb24-4916-b915-a2878e2686b6" containerName="maas-api" containerID="cri-o://5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e" gracePeriod=30
Oct 30 15:38:02.895907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.895875    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:38:02.896202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.896009    5064 generic.go:356] "Generic (PLEG): container finished" podID="3c51e121-fb24-4916-b915-a2878e2686b6" containerID="5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e" exitCode=0
Oct 30 15:38:02.896435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.896030    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" event={"ID":"3c51e121-fb24-4916-b915-a2878e2686b6","Type":"ContainerDied","Data":"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"}
Oct 30 15:38:02.896578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.896558    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb" event={"ID":"3c51e121-fb24-4916-b915-a2878e2686b6","Type":"ContainerDied","Data":"e0401fa9be53fe5a18d2217c42dac725ec08577e817e913beeb5d47ad552ea5c"}
Oct 30 15:38:02.896720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.896646    5064 scope.go:122] "RemoveContainer" containerID="5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"
Oct 30 15:38:02.913923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.913900    5064 scope.go:122] "RemoveContainer" containerID="5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"
Oct 30 15:38:02.914303 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:02.914275    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e\": container with ID starting with 5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e not found: ID does not exist" containerID="5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"
Oct 30 15:38:02.914387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:02.914317    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e"} err="failed to get container status \"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e\": rpc error: code = NotFound desc = could not find container \"5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e\": container with ID starting with 5efd3628b60e6854b045a7c3f870ab60632771fb63081c5e1dc03c88238bc99e not found: ID does not exist"
Oct 30 15:38:03.014895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.014856    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\") pod \"3c51e121-fb24-4916-b915-a2878e2686b6\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") "
Oct 30 15:38:03.015108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.014954    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\") pod \"3c51e121-fb24-4916-b915-a2878e2686b6\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") "
Oct 30 15:38:03.015321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.015289    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca" pod "3c51e121-fb24-4916-b915-a2878e2686b6" (UID: "3c51e121-fb24-4916-b915-a2878e2686b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:03.015473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.015451    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\" (UniqueName: \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\") pod \"3c51e121-fb24-4916-b915-a2878e2686b6\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") "
Oct 30 15:38:03.015983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.015954    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\") pod \"3c51e121-fb24-4916-b915-a2878e2686b6\" (UID: \"3c51e121-fb24-4916-b915-a2878e2686b6\") "
Oct 30 15:38:03.016738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.016712    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c51e121-fb24-4916-b915-a2878e2686b6-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:03.018027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.017996    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs" pod "3c51e121-fb24-4916-b915-a2878e2686b6" (UID: "3c51e121-fb24-4916-b915-a2878e2686b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:03.018327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.018306    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm" pod "3c51e121-fb24-4916-b915-a2878e2686b6" (UID: "3c51e121-fb24-4916-b915-a2878e2686b6"). InnerVolumeSpecName "kube-api-access-t4cgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:03.019544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.019523    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls" pod "3c51e121-fb24-4916-b915-a2878e2686b6" (UID: "3c51e121-fb24-4916-b915-a2878e2686b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:03.119066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.119028    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:03.119066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.119067    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c51e121-fb24-4916-b915-a2878e2686b6-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:03.119473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.119081    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cgm\" (UniqueName: \"kubernetes.io/projected/3c51e121-fb24-4916-b915-a2878e2686b6-kube-api-access-t4cgm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:03.905951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.905923    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"
Oct 30 15:38:03.966998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.966956    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:38:03.971021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:03.970991    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-985bf6-864c4f56f5-77xsb"]
Oct 30 15:38:04.794175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:04.794138    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c51e121-fb24-4916-b915-a2878e2686b6" path="/var/lib/kubelet/pods/3c51e121-fb24-4916-b915-a2878e2686b6/volumes"
Oct 30 15:38:04.973325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:04.973291    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 15:38:04.973589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:04.973551    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" podUID="8f7bef0f-cc45-4218-81cb-838abf3f111e" containerName="maas-api" containerID="cri-o://1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9" gracePeriod=30
Oct 30 15:38:05.723421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.723336    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 15:38:05.793436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.793364    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\") pod \"8f7bef0f-cc45-4218-81cb-838abf3f111e\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") "
Oct 30 15:38:05.793628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.793523    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") pod \"8f7bef0f-cc45-4218-81cb-838abf3f111e\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") "
Oct 30 15:38:05.794602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.794578    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\") pod \"8f7bef0f-cc45-4218-81cb-838abf3f111e\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") "
Oct 30 15:38:05.795063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.794630    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\" (UniqueName: \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\") pod \"8f7bef0f-cc45-4218-81cb-838abf3f111e\" (UID: \"8f7bef0f-cc45-4218-81cb-838abf3f111e\") "
Oct 30 15:38:05.798214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.798166    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca" pod "8f7bef0f-cc45-4218-81cb-838abf3f111e" (UID: "8f7bef0f-cc45-4218-81cb-838abf3f111e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:05.811469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.805068    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7" pod "8f7bef0f-cc45-4218-81cb-838abf3f111e" (UID: "8f7bef0f-cc45-4218-81cb-838abf3f111e"). InnerVolumeSpecName "kube-api-access-ftpg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:05.811865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.811816    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs" pod "8f7bef0f-cc45-4218-81cb-838abf3f111e" (UID: "8f7bef0f-cc45-4218-81cb-838abf3f111e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:05.830143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.830070    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls" pod "8f7bef0f-cc45-4218-81cb-838abf3f111e" (UID: "8f7bef0f-cc45-4218-81cb-838abf3f111e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:05.897210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.897172    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:05.897210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.897207    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7bef0f-cc45-4218-81cb-838abf3f111e-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:05.897487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.897221    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7bef0f-cc45-4218-81cb-838abf3f111e-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:05.897487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.897234    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpg7\" (UniqueName: \"kubernetes.io/projected/8f7bef0f-cc45-4218-81cb-838abf3f111e-kube-api-access-ftpg7\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:05.924617 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.924583    5064 generic.go:356] "Generic (PLEG): container finished" podID="8f7bef0f-cc45-4218-81cb-838abf3f111e" containerID="1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9" exitCode=0
Oct 30 15:38:05.924821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.924639    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"
Oct 30 15:38:05.924998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.924635    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" event={"ID":"8f7bef0f-cc45-4218-81cb-838abf3f111e","Type":"ContainerDied","Data":"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"}
Oct 30 15:38:05.924998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.924983    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l" event={"ID":"8f7bef0f-cc45-4218-81cb-838abf3f111e","Type":"ContainerDied","Data":"c63f6541e38cfae4e3950c8c0e89f9c9b1ad4c7f4bc0c0115239542d6cf3b642"}
Oct 30 15:38:05.925124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.925015    5064 scope.go:122] "RemoveContainer" containerID="1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"
Oct 30 15:38:05.939905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.939877    5064 scope.go:122] "RemoveContainer" containerID="1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"
Oct 30 15:38:05.940289 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:05.940260    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9\": container with ID starting with 1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9 not found: ID does not exist" containerID="1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"
Oct 30 15:38:05.940397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.940299    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9"} err="failed to get container status \"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9\": rpc error: code = NotFound desc = could not find container \"1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9\": container with ID starting with 1dd013e33311390dfc6b59fd22346a3289e9de824da73f8dc2a678fb01f033e9 not found: ID does not exist"
Oct 30 15:38:05.989870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.989769    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 15:38:05.993967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:05.993935    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-4f5c30-5df687456-mdd2l"]
Oct 30 15:38:06.729969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:06.728576    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7bef0f-cc45-4218-81cb-838abf3f111e" path="/var/lib/kubelet/pods/8f7bef0f-cc45-4218-81cb-838abf3f111e/volumes"
Oct 30 15:38:10.780090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:10.780059    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:38:10.780526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:10.780503    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:38:10.780726 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:10.780701    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy" containerID="cri-o://d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44" gracePeriod=30
Oct 30 15:38:10.780988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:10.780963    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" containerID="cri-o://82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101" gracePeriod=30
Oct 30 15:38:12.064454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:12.064409    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:38:12.064909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:12.064483    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:38:13.869707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:13.868562    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 15:38:13.869707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:13.868776    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy" containerID="cri-o://fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3" gracePeriod=30
Oct 30 15:38:16.123082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.122593    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:38:16.138669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.138631    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.138898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.138682    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.138898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.138771    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.138898 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.138805    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.139107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.138984    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.139233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.139212    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.139290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.139270    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.139353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.139314    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.139413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.139349    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\") pod \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\" (UID: \"a4bb4726-db4d-42f9-abd4-8bfcd6af938a\") "
Oct 30 15:38:16.140877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.140815    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.141010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.140956    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.141222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.141196    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:16.142169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.142141    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.143261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.143233    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:38:16.144143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.144121    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "kube-api-access-jwj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:16.145108 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.144946    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.145214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.145180    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:16.146327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.146302    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy" pod "a4bb4726-db4d-42f9-abd4-8bfcd6af938a" (UID: "a4bb4726-db4d-42f9-abd4-8bfcd6af938a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.178357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.178322    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:38:16.222310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.222270    5064 generic.go:356] "Generic (PLEG): container finished" podID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerID="82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101" exitCode=0
Oct 30 15:38:16.222310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.222300    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" event={"ID":"e0ad6564-7674-4295-b749-17cbcde1faf1","Type":"ContainerDied","Data":"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"}
Oct 30 15:38:16.222569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.222325    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"
Oct 30 15:38:16.222569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.222390    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g" event={"ID":"e0ad6564-7674-4295-b749-17cbcde1faf1","Type":"ContainerDied","Data":"a51d7f66f1ae3e5163dd8d1123c062e382a5d2ce758de3bb381c9d13bba0821b"}
Oct 30 15:38:16.222569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.222420    5064 scope.go:122] "RemoveContainer" containerID="82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"
Oct 30 15:38:16.224468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.224440    5064 generic.go:356] "Generic (PLEG): container finished" podID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerID="d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44" exitCode=0
Oct 30 15:38:16.224639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.224621    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerDied","Data":"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"}
Oct 30 15:38:16.224703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.224672    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl" event={"ID":"a4bb4726-db4d-42f9-abd4-8bfcd6af938a","Type":"ContainerDied","Data":"c8faa17c82250ec1a433a5982aa0feebe1cdb552c967a127d5158419debddfce"}
Oct 30 15:38:16.224784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.224771    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"
Oct 30 15:38:16.235041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.235010    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:38:16.235189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.235082    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:38:16.238074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.238054    5064 scope.go:122] "RemoveContainer" containerID="82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"
Oct 30 15:38:16.238394 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:16.238368    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101\": container with ID starting with 82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101 not found: ID does not exist" containerID="82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"
Oct 30 15:38:16.238476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.238407    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101"} err="failed to get container status \"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101\": rpc error: code = NotFound desc = could not find container \"82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101\": container with ID starting with 82f4f3509f5829861932b4469adcfbdc5eed12a82fe1b0939f3b5a7bd0d3c101 not found: ID does not exist"
Oct 30 15:38:16.238476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.238431    5064 scope.go:122] "RemoveContainer" containerID="d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"
Oct 30 15:38:16.241069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241039    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241155    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241196    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241312 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241278    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241310    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241401 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241342    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241397    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241447    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.241526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.241476    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\") pod \"e0ad6564-7674-4295-b749-17cbcde1faf1\" (UID: \"e0ad6564-7674-4295-b749-17cbcde1faf1\") "
Oct 30 15:38:16.242460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242432    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242462    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242478    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242493    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242506    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242519    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242530    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242543    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.242569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.242555    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwj75\" (UniqueName: \"kubernetes.io/projected/a4bb4726-db4d-42f9-abd4-8bfcd6af938a-kube-api-access-jwj75\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.244281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.244015    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.244281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.244199    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.244394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.244361    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.244667 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.244629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.247435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.244977    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:16.258995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.258958    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:16.260406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.260382    5064 scope.go:122] "RemoveContainer" containerID="66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"
Oct 30 15:38:16.266202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.266175    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:38:16.266202 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.266195    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:16.266339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.266176    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q" pod "e0ad6564-7674-4295-b749-17cbcde1faf1" (UID: "e0ad6564-7674-4295-b749-17cbcde1faf1"). InnerVolumeSpecName "kube-api-access-ctl8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:16.272764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.272743    5064 scope.go:122] "RemoveContainer" containerID="d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"
Oct 30 15:38:16.273203 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:16.273166    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44\": container with ID starting with d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44 not found: ID does not exist" containerID="d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"
Oct 30 15:38:16.273324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.273282    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44"} err="failed to get container status \"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44\": rpc error: code = NotFound desc = could not find container \"d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44\": container with ID starting with d30b2b70795810c9e102379ad4baf5cad3882cbbbfed76e312febe4851092e44 not found: ID does not exist"
Oct 30 15:38:16.273324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.273314    5064 scope.go:122] "RemoveContainer" containerID="66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"
Oct 30 15:38:16.273684 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:16.273642    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1\": container with ID starting with 66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1 not found: ID does not exist" containerID="66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"
Oct 30 15:38:16.273800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.273678    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1"} err="failed to get container status \"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1\": rpc error: code = NotFound desc = could not find container \"66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1\": container with ID starting with 66096f34bb76f1a9df69ca9e39930aa4ceee5b278dfa7e6d58741c816ec36ae1 not found: ID does not exist"
Oct 30 15:38:16.292045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.292006    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:38:16.300060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.300024    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-5kbjl"]
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344161    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344194    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344204    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344212    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344223    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctl8q\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-kube-api-access-ctl8q\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344232    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344245    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad6564-7674-4295-b749-17cbcde1faf1-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344255    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.344816 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.344262    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad6564-7674-4295-b749-17cbcde1faf1-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:16.580805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.580766    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:38:16.584736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.584708    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-jhk6g"]
Oct 30 15:38:16.726187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.726152    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" path="/var/lib/kubelet/pods/a4bb4726-db4d-42f9-abd4-8bfcd6af938a/volumes"
Oct 30 15:38:16.726845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:16.726812    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" path="/var/lib/kubelet/pods/e0ad6564-7674-4295-b749-17cbcde1faf1/volumes"
Oct 30 15:38:19.184078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.184043    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 15:38:19.257919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.257809    5064 generic.go:356] "Generic (PLEG): container finished" podID="9905b36f-8be0-460c-abc4-acd7404396ba" containerID="fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3" exitCode=0
Oct 30 15:38:19.257919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.257884    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"
Oct 30 15:38:19.257919 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.257900    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerDied","Data":"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"}
Oct 30 15:38:19.258216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.257973    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw" event={"ID":"9905b36f-8be0-460c-abc4-acd7404396ba","Type":"ContainerDied","Data":"a5fc40c4025429ff98dc247c981e0cc1ab16f5d3b6a4bf96b60d6a25aee8ee17"}
Oct 30 15:38:19.258216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.258013    5064 scope.go:122] "RemoveContainer" containerID="fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"
Oct 30 15:38:19.263528 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.263493    5064 generic.go:356] "Generic (PLEG): container finished" podID="7176ec41-5592-4f76-b284-203843247cf8" containerID="e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555" exitCode=0
Oct 30 15:38:19.263681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.263615    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerDied","Data":"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"}
Oct 30 15:38:19.264181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.264160    5064 scope.go:122] "RemoveContainer" containerID="e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"
Oct 30 15:38:19.266472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.266440    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:19.266587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.266515    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:19.266587 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.266549    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:19.293116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.293054    5064 scope.go:122] "RemoveContainer" containerID="fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"
Oct 30 15:38:19.313307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.313279    5064 scope.go:122] "RemoveContainer" containerID="fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"
Oct 30 15:38:19.314996 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:19.314960    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3\": container with ID starting with fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3 not found: ID does not exist" containerID="fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"
Oct 30 15:38:19.315127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.315008    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3"} err="failed to get container status \"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3\": rpc error: code = NotFound desc = could not find container \"fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3\": container with ID starting with fbe9519f4763a78948af33c1bad46960a586bc95c1927d73847a31caa82d69a3 not found: ID does not exist"
Oct 30 15:38:19.315127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.315032    5064 scope.go:122] "RemoveContainer" containerID="fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"
Oct 30 15:38:19.315357 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:19.315334    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58\": container with ID starting with fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58 not found: ID does not exist" containerID="fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"
Oct 30 15:38:19.315492 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.315362    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58"} err="failed to get container status \"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58\": rpc error: code = NotFound desc = could not find container \"fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58\": container with ID starting with fd7ae62216d25269aeb1d3e6245c3471317845f5b4bbcf7fecc7c638d4458c58 not found: ID does not exist"
Oct 30 15:38:19.346548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346508    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346592    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346664    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346800 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346770    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346821    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346869    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.346996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346905    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.347152 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.346996    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.347152 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.347062    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\") pod \"9905b36f-8be0-460c-abc4-acd7404396ba\" (UID: \"9905b36f-8be0-460c-abc4-acd7404396ba\") "
Oct 30 15:38:19.347981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.347948    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:19.348858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.348812    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.349008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.348981    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:19.349423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.349363    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:19.349546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.349407    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:19.349911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.349889    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:19.350316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.350289    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:19.352879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.352786    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:19.361569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.361042    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:38:19.361569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.361199    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f" pod "9905b36f-8be0-460c-abc4-acd7404396ba" (UID: "9905b36f-8be0-460c-abc4-acd7404396ba"). InnerVolumeSpecName "kube-api-access-7pj4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:19.450765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450726    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.450765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450759    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.450765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450769    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.451022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450777    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905b36f-8be0-460c-abc4-acd7404396ba-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.451022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450786    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905b36f-8be0-460c-abc4-acd7404396ba-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.451022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450793    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905b36f-8be0-460c-abc4-acd7404396ba-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.451022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450800    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.451022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.450809    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pj4f\" (UniqueName: \"kubernetes.io/projected/9905b36f-8be0-460c-abc4-acd7404396ba-kube-api-access-7pj4f\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:19.611274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.611234    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 15:38:19.614864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:19.614820    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-cfprw"]
Oct 30 15:38:20.274887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:20.274753    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerStarted","Data":"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"}
Oct 30 15:38:20.412652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:20.412613    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:38:20.412872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:20.412815    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:20.412959 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:20.412887    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:20.748647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:20.748612    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" path="/var/lib/kubelet/pods/9905b36f-8be0-460c-abc4-acd7404396ba/volumes"
Oct 30 15:38:21.412205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:21.412165    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:21.412636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:21.412240    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:22.412633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:22.412590    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:22.432070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:22.432024    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:23.412279 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:23.412235    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:23.412469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:23.412308    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:24.424105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:24.423262    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:24.424105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:24.423331    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:24.955440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:24.955402    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:38:24.955679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:24.955638    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" podUID="4a30de2c-790a-4a13-855f-5934160ea19c" containerName="maas-api" containerID="cri-o://9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3" gracePeriod=30
Oct 30 15:38:25.301214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.300866    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:38:25.362173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.362138    5064 generic.go:356] "Generic (PLEG): container finished" podID="4a30de2c-790a-4a13-855f-5934160ea19c" containerID="9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3" exitCode=0
Oct 30 15:38:25.362384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.362226    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" event={"ID":"4a30de2c-790a-4a13-855f-5934160ea19c","Type":"ContainerDied","Data":"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"}
Oct 30 15:38:25.362384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.362283    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"
Oct 30 15:38:25.362384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.362298    5064 scope.go:122] "RemoveContainer" containerID="9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"
Oct 30 15:38:25.362592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.362284    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb" event={"ID":"4a30de2c-790a-4a13-855f-5934160ea19c","Type":"ContainerDied","Data":"3526f6e907f4b6b15787248e951bd268193d2b14cc798f334a12dc9d48f6dccc"}
Oct 30 15:38:25.368511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.368479    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\") pod \"4a30de2c-790a-4a13-855f-5934160ea19c\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") "
Oct 30 15:38:25.368655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.368528    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\" (UniqueName: \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\") pod \"4a30de2c-790a-4a13-855f-5934160ea19c\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") "
Oct 30 15:38:25.368655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.368627    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\") pod \"4a30de2c-790a-4a13-855f-5934160ea19c\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") "
Oct 30 15:38:25.368823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.368802    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\") pod \"4a30de2c-790a-4a13-855f-5934160ea19c\" (UID: \"4a30de2c-790a-4a13-855f-5934160ea19c\") "
Oct 30 15:38:25.371771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.371738    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs" pod "4a30de2c-790a-4a13-855f-5934160ea19c" (UID: "4a30de2c-790a-4a13-855f-5934160ea19c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:25.372103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.372078    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca" pod "4a30de2c-790a-4a13-855f-5934160ea19c" (UID: "4a30de2c-790a-4a13-855f-5934160ea19c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:25.374435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.374410    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755" pod "4a30de2c-790a-4a13-855f-5934160ea19c" (UID: "4a30de2c-790a-4a13-855f-5934160ea19c"). InnerVolumeSpecName "kube-api-access-tj755". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:25.376738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.376720    5064 scope.go:122] "RemoveContainer" containerID="9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"
Oct 30 15:38:25.377046 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:25.377023    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3\": container with ID starting with 9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3 not found: ID does not exist" containerID="9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"
Oct 30 15:38:25.377141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.377057    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3"} err="failed to get container status \"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3\": rpc error: code = NotFound desc = could not find container \"9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3\": container with ID starting with 9c898a135fde14e1176ab7f6354f93e194ea0a46f5790f5d57010b452953e8d3 not found: ID does not exist"
Oct 30 15:38:25.377216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.377173    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls" pod "4a30de2c-790a-4a13-855f-5934160ea19c" (UID: "4a30de2c-790a-4a13-855f-5934160ea19c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:25.418651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.417852    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 15:38:25.418651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.417917    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 15:38:25.471666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.471625    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30de2c-790a-4a13-855f-5934160ea19c-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:25.471666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.471666    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj755\" (UniqueName: \"kubernetes.io/projected/4a30de2c-790a-4a13-855f-5934160ea19c-kube-api-access-tj755\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:25.472239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.471683    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:25.472239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.471697    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a30de2c-790a-4a13-855f-5934160ea19c-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:25.745366 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.745325    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:38:25.749145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:25.749114    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78b513-f7c94bcf5-t9dkb"]
Oct 30 15:38:26.418041 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:26.418003    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:38:26.418321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:26.418290    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:38:26.418956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:26.418935    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:38:26.727506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:26.727404    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a30de2c-790a-4a13-855f-5934160ea19c" path="/var/lib/kubelet/pods/4a30de2c-790a-4a13-855f-5934160ea19c/volumes"
Oct 30 15:38:29.668647 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:29.668613    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:38:40.112014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.111971    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:38:40.112546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.112230    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" podUID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" containerName="maas-api" containerID="cri-o://b756a80b856c4905dc1df27d20fb26b7f14c4de42ab00f39c25af175bb9fd58a" gracePeriod=30
Oct 30 15:38:40.525865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.521513    5064 generic.go:356] "Generic (PLEG): container finished" podID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" containerID="b756a80b856c4905dc1df27d20fb26b7f14c4de42ab00f39c25af175bb9fd58a" exitCode=0
Oct 30 15:38:40.525865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.521718    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" event={"ID":"20a233db-99b1-4dfb-91fb-5a818a3c4c1a","Type":"ContainerDied","Data":"b756a80b856c4905dc1df27d20fb26b7f14c4de42ab00f39c25af175bb9fd58a"}
Oct 30 15:38:40.525865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.521768    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92" event={"ID":"20a233db-99b1-4dfb-91fb-5a818a3c4c1a","Type":"ContainerDied","Data":"55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557"}
Oct 30 15:38:40.525865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.521794    5064 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c3b6bcda19224c5b2e706178c141f8da37667d6eb17011c2094f605ccb1557"
Oct 30 15:38:40.597856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.593223    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:38:40.670853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.668495    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\" (UniqueName: \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\") pod \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") "
Oct 30 15:38:40.670853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.668668    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\") pod \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") "
Oct 30 15:38:40.670853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.668783    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\") pod \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") "
Oct 30 15:38:40.670853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.668815    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\") pod \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\" (UID: \"20a233db-99b1-4dfb-91fb-5a818a3c4c1a\") "
Oct 30 15:38:40.670853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.669740    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca" pod "20a233db-99b1-4dfb-91fb-5a818a3c4c1a" (UID: "20a233db-99b1-4dfb-91fb-5a818a3c4c1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:40.685605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.671546    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:40.697367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.696477    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6" pod "20a233db-99b1-4dfb-91fb-5a818a3c4c1a" (UID: "20a233db-99b1-4dfb-91fb-5a818a3c4c1a"). InnerVolumeSpecName "kube-api-access-h5mf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:40.720308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.709167    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls" pod "20a233db-99b1-4dfb-91fb-5a818a3c4c1a" (UID: "20a233db-99b1-4dfb-91fb-5a818a3c4c1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:40.720308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.710286    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs" pod "20a233db-99b1-4dfb-91fb-5a818a3c4c1a" (UID: "20a233db-99b1-4dfb-91fb-5a818a3c4c1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:40.776692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.776569    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:40.776692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.776615    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:40.776692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:40.776639    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mf6\" (UniqueName: \"kubernetes.io/projected/20a233db-99b1-4dfb-91fb-5a818a3c4c1a-kube-api-access-h5mf6\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:41.528369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:41.528336    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"
Oct 30 15:38:41.588643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:41.588608    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:38:41.601701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:41.601666    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-609709-5f886b789-zjb92"]
Oct 30 15:38:42.725186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:42.725150    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" path="/var/lib/kubelet/pods/20a233db-99b1-4dfb-91fb-5a818a3c4c1a/volumes"
Oct 30 15:38:45.135405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.135366    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:38:45.135925 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.135619    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" podUID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" containerName="maas-api" containerID="cri-o://68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60" gracePeriod=30
Oct 30 15:38:45.452803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.452773    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:38:45.470277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.470248    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\" (UniqueName: \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\") pod \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") "
Oct 30 15:38:45.470459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.470413    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") pod \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") "
Oct 30 15:38:45.470459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.470452    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\") pod \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") "
Oct 30 15:38:45.470578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.470477    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\") pod \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\" (UID: \"92a3593c-2d8a-4b90-ad42-ee87b0274f71\") "
Oct 30 15:38:45.470913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.470876    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca" pod "92a3593c-2d8a-4b90-ad42-ee87b0274f71" (UID: "92a3593c-2d8a-4b90-ad42-ee87b0274f71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:45.471350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.471315    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a3593c-2d8a-4b90-ad42-ee87b0274f71-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:45.472592 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.472564    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl" pod "92a3593c-2d8a-4b90-ad42-ee87b0274f71" (UID: "92a3593c-2d8a-4b90-ad42-ee87b0274f71"). InnerVolumeSpecName "kube-api-access-cdzwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:45.472718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.472638    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs" pod "92a3593c-2d8a-4b90-ad42-ee87b0274f71" (UID: "92a3593c-2d8a-4b90-ad42-ee87b0274f71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:45.473080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.473057    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls" pod "92a3593c-2d8a-4b90-ad42-ee87b0274f71" (UID: "92a3593c-2d8a-4b90-ad42-ee87b0274f71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:38:45.561133 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.561087    5064 generic.go:356] "Generic (PLEG): container finished" podID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" containerID="68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60" exitCode=0
Oct 30 15:38:45.561339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.561146    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"
Oct 30 15:38:45.561339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.561312    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" event={"ID":"92a3593c-2d8a-4b90-ad42-ee87b0274f71","Type":"ContainerDied","Data":"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"}
Oct 30 15:38:45.561466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.561353    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq" event={"ID":"92a3593c-2d8a-4b90-ad42-ee87b0274f71","Type":"ContainerDied","Data":"dccb8248eb11b5bf3791d5d6caae5416db069c0393160bee48d4fe1fe95e8630"}
Oct 30 15:38:45.561466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.561379    5064 scope.go:122] "RemoveContainer" containerID="68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"
Oct 30 15:38:45.572550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.572520    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzwl\" (UniqueName: \"kubernetes.io/projected/92a3593c-2d8a-4b90-ad42-ee87b0274f71-kube-api-access-cdzwl\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:45.572550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.572547    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:45.572718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.572564    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a3593c-2d8a-4b90-ad42-ee87b0274f71-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:45.573700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.573682    5064 scope.go:122] "RemoveContainer" containerID="68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"
Oct 30 15:38:45.573983 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:45.573963    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60\": container with ID starting with 68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60 not found: ID does not exist" containerID="68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"
Oct 30 15:38:45.574057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.573996    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60"} err="failed to get container status \"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60\": rpc error: code = NotFound desc = could not find container \"68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60\": container with ID starting with 68100073c170f126a427cf62bb0a05919c908fa21ba9a77ea34fdc556dc3bc60 not found: ID does not exist"
Oct 30 15:38:45.640480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.640426    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:38:45.674493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:45.673589    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-908638-85b7c6dbdb-4mcgq"]
Oct 30 15:38:46.724822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:46.724786    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" path="/var/lib/kubelet/pods/92a3593c-2d8a-4b90-ad42-ee87b0274f71/volumes"
Oct 30 15:38:49.708274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:49.708236    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:38:49.709030 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:49.708995    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy" containerID="cri-o://2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e" gracePeriod=30
Oct 30 15:38:53.796783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:53.796738    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:38:53.797362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:53.797045    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy" containerID="cri-o://2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095" gracePeriod=30
Oct 30 15:38:54.014503 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.014460    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015632    5064 cpu_manager.go:414] "containerMap: removing container" podUID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" containerName="maas-api"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015654    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" containerName="maas-api"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015687    5064 cpu_manager.go:414] "containerMap: removing container" podUID="4a30de2c-790a-4a13-855f-5934160ea19c" containerName="maas-api"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015695    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a30de2c-790a-4a13-855f-5934160ea19c" containerName="maas-api"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015709    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015718    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015734    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.015743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015742    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015761    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8f7bef0f-cc45-4218-81cb-838abf3f111e" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015768    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7bef0f-cc45-4218-81cb-838abf3f111e" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015795    5064 cpu_manager.go:414] "containerMap: removing container" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015802    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015849    5064 cpu_manager.go:414] "containerMap: removing container" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015858    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015883    5064 cpu_manager.go:414] "containerMap: removing container" podUID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015890    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015903    5064 cpu_manager.go:414] "containerMap: removing container" podUID="3c51e121-fb24-4916-b915-a2878e2686b6" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015910    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c51e121-fb24-4916-b915-a2878e2686b6" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015942    5064 cpu_manager.go:414] "containerMap: removing container" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015949    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015967    5064 cpu_manager.go:414] "containerMap: removing container" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.015973    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016154    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a30de2c-790a-4a13-855f-5934160ea19c" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016167    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016176    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bb4726-db4d-42f9-abd4-8bfcd6af938a" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016187    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7bef0f-cc45-4218-81cb-838abf3f111e" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016202    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016213    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c3edf-e93b-484d-ac7a-6dc6f91d8505" containerName="payload-pre-processing"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016226    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3593c-2d8a-4b90-ad42-ee87b0274f71" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016238    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c51e121-fb24-4916-b915-a2878e2686b6" containerName="maas-api"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016253    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ad6564-7674-4295-b749-17cbcde1faf1" containerName="istio-proxy"
Oct 30 15:38:54.016308 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.016262    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a233db-99b1-4dfb-91fb-5a818a3c4c1a" containerName="maas-api"
Oct 30 15:38:54.022255 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.022234    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.048012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.047927    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-01c89d-openshift-default-dockercfg-mvc5p\""
Oct 30 15:38:54.048625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.048590    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:38:54.075703 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.075657    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.075983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.075761    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.075983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.075889    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.076135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.075994    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.076597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.076569    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.076746 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.076716    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdrb\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.076904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.076876    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.076988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.076920    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.077098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.077067    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.178460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178422    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.178679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178558    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.178679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178595    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.178679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178635    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178695    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.178739    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179115    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179153    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179284    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179298    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdrb\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179417    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179455    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.179875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.179851    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.180091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.180069    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.181072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.181050    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.181585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.181565    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.229563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.229534    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.230853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.230812    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdrb\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\") pod \"e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") " pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.390855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.384130    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:54.979014 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.978924    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:38:54.984462 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:38:54.984421    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b59509_9b9a_4399_9f13_4453158404e4.slice/crio-3fcd440645a8d5f797e753721e516733ae1cec90ac75c1af7c1358553a5b349c WatchSource:0}: Error finding container 3fcd440645a8d5f797e753721e516733ae1cec90ac75c1af7c1358553a5b349c: Status 404 returned error can't find the container with id 3fcd440645a8d5f797e753721e516733ae1cec90ac75c1af7c1358553a5b349c
Oct 30 15:38:54.988869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.988797    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:54.988989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.988935    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:54.989035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:54.989012    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:38:55.001485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.001457    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:38:55.007049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007024    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007062    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007089    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007112    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007131    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007154    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007266    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007428    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:55.007486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007456    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:55.007656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007483    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007578    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\") pod \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\" (UID: \"9f0259dc-47bb-41a7-976a-3d56d504bfdc\") "
Oct 30 15:38:55.007767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007711    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:55.007821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007765    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:55.007821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.007776    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:55.008380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.008343    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.008380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.008381    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.008565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.008391    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.008565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.008400    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.008565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.008410    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.009391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.009368    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:55.009815 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.009790    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:55.009911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.009839    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:38:55.010716 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.010693    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc" pod "9f0259dc-47bb-41a7-976a-3d56d504bfdc" (UID: "9f0259dc-47bb-41a7-976a-3d56d504bfdc"). InnerVolumeSpecName "kube-api-access-vnwvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:55.109763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.109714    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.109763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.109758    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwvc\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-kube-api-access-vnwvc\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.109763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.109771    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.110135 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.109779    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0259dc-47bb-41a7-976a-3d56d504bfdc-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:55.679573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.679531    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerStarted","Data":"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"}
Oct 30 15:38:55.679812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.679600    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerStarted","Data":"3fcd440645a8d5f797e753721e516733ae1cec90ac75c1af7c1358553a5b349c"}
Oct 30 15:38:55.682031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.682002    5064 generic.go:356] "Generic (PLEG): container finished" podID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerID="2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e" exitCode=0
Oct 30 15:38:55.682171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.682048    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"
Oct 30 15:38:55.682171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.682081    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerDied","Data":"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"}
Oct 30 15:38:55.682171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.682132    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh" event={"ID":"9f0259dc-47bb-41a7-976a-3d56d504bfdc","Type":"ContainerDied","Data":"2db0d516f0d2da5db9f906eb752794c431bda11d79e77f9e34acfefdc4eec28f"}
Oct 30 15:38:55.682171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.682154    5064 scope.go:122] "RemoveContainer" containerID="2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"
Oct 30 15:38:55.693803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.693781    5064 scope.go:122] "RemoveContainer" containerID="fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"
Oct 30 15:38:55.703822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.703803    5064 scope.go:122] "RemoveContainer" containerID="2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"
Oct 30 15:38:55.704117 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:55.704095    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e\": container with ID starting with 2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e not found: ID does not exist" containerID="2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"
Oct 30 15:38:55.704200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.704127    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e"} err="failed to get container status \"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e\": rpc error: code = NotFound desc = could not find container \"2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e\": container with ID starting with 2385a93c61c9d0bafc7a376294c730bdebed6f82c220834af83b23e78dedf56e not found: ID does not exist"
Oct 30 15:38:55.704200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.704147    5064 scope.go:122] "RemoveContainer" containerID="fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"
Oct 30 15:38:55.704451 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:55.704430    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3\": container with ID starting with fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3 not found: ID does not exist" containerID="fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"
Oct 30 15:38:55.704497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.704459    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3"} err="failed to get container status \"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3\": rpc error: code = NotFound desc = could not find container \"fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3\": container with ID starting with fd0713241e8ba2ee0e301c301306911eee1856fc8ad0b574306f7afa31d400b3 not found: ID does not exist"
Oct 30 15:38:55.804654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.804586    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podStartSLOduration=2.804564139 podStartE2EDuration="2.804564139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:55.783583193 +0000 UTC m=+3085.225473913" watchObservedRunningTime="2026-10-30 15:38:55.804564139 +0000 UTC m=+3085.246454857"
Oct 30 15:38:55.829204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.829173    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:38:55.840151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:55.840121    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-wmsxh"]
Oct 30 15:38:56.245743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.245703    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:38:56.246239 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.245936    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy" containerID="cri-o://96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07" gracePeriod=30
Oct 30 15:38:56.250475 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.250449    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:38:56.250769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.250742    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy" containerID="cri-o://37e063e38ccde8d356b01e40567faffabe7ce8304b5807a69b0f2981bddc164f" gracePeriod=30
Oct 30 15:38:56.385661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.385628    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:56.392362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.392330    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:56.692978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.692944    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:56.693899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.693874    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:38:56.725000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:56.724967    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" path="/var/lib/kubelet/pods/9f0259dc-47bb-41a7-976a-3d56d504bfdc/volumes"
Oct 30 15:38:59.082373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.082305    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:38:59.217562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217515    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.217562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217560    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.217824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217631    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.217824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217680    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.217824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217721    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.218020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217875    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.218020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217883    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:59.218020 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.217919    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.218171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218037    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.218171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218102    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:38:59.218171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218106    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:59.218171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218125    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\") pod \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\" (UID: \"08ca11fd-24f5-42f8-85ae-125c19ffaba5\") "
Oct 30 15:38:59.218171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218139    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:59.218423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218235    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:59.218902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218875    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.218902 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218903    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.219153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218918    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.219153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218933    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.219153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.218947    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.220524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.220495    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:38:59.220662 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.220640    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:59.221300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.221279    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "kube-api-access-82r2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:38:59.221429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.221393    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo" pod "08ca11fd-24f5-42f8-85ae-125c19ffaba5" (UID: "08ca11fd-24f5-42f8-85ae-125c19ffaba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:38:59.325298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.325255    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82r2v\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-kube-api-access-82r2v\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.325298 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.325298    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.325555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.325315    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.325555 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.325330    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ca11fd-24f5-42f8-85ae-125c19ffaba5-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:38:59.638264 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:59.638229    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:38:59.717795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.717757    5064 generic.go:356] "Generic (PLEG): container finished" podID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerID="2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095" exitCode=0
Oct 30 15:38:59.718069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.717800    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"
Oct 30 15:38:59.718069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.717881    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerDied","Data":"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"}
Oct 30 15:38:59.718069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.717935    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt" event={"ID":"08ca11fd-24f5-42f8-85ae-125c19ffaba5","Type":"ContainerDied","Data":"452ebf2536ff0642e37896e6108a3e49699fdf5c83dc9dc784c44509d0bbec93"}
Oct 30 15:38:59.718069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.717958    5064 scope.go:122] "RemoveContainer" containerID="2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"
Oct 30 15:38:59.731467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.731444    5064 scope.go:122] "RemoveContainer" containerID="99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"
Oct 30 15:38:59.741819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.741800    5064 scope.go:122] "RemoveContainer" containerID="2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"
Oct 30 15:38:59.742145 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:59.742122    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095\": container with ID starting with 2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095 not found: ID does not exist" containerID="2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"
Oct 30 15:38:59.742214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.742154    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095"} err="failed to get container status \"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095\": rpc error: code = NotFound desc = could not find container \"2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095\": container with ID starting with 2d9708ece6e364fcc92dbc4c19eb47cf939927841d7aba4d774094c8afcf4095 not found: ID does not exist"
Oct 30 15:38:59.742214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.742172    5064 scope.go:122] "RemoveContainer" containerID="99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"
Oct 30 15:38:59.742435 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:38:59.742417    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914\": container with ID starting with 99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914 not found: ID does not exist" containerID="99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"
Oct 30 15:38:59.742494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.742441    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914"} err="failed to get container status \"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914\": rpc error: code = NotFound desc = could not find container \"99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914\": container with ID starting with 99f819579e6996b7bd3404a73bc707b767e522997c18c7f15ef6bde5e4782914 not found: ID does not exist"
Oct 30 15:38:59.786220 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.786187    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:38:59.789695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.789672    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-908638-openshift-default-59cb7c664f-hmhxt"]
Oct 30 15:38:59.987713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.987101    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988341    5064 cpu_manager.go:414] "containerMap: removing container" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988372    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988424    5064 cpu_manager.go:414] "containerMap: removing container" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988433    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988475    5064 cpu_manager.go:414] "containerMap: removing container" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988484    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988534    5064 cpu_manager.go:414] "containerMap: removing container" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988541    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988693    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988709    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="9905b36f-8be0-460c-abc4-acd7404396ba" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988722    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:38:59.992952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.988737    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0259dc-47bb-41a7-976a-3d56d504bfdc" containerName="istio-proxy"
Oct 30 15:39:00.007864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:38:59.999132    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.024300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.017205    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-261ad0-dockercfg-7lw2z\""
Oct 30 15:39:00.024300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.017954    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-261ad0\""
Oct 30 15:39:00.024300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.020172    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:39:00.145946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.145889    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4gf\" (UniqueName: \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.146449 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.146165    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.148221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.148192    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:39:00.149151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.149129    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" containerName="istio-proxy"
Oct 30 15:39:00.153037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.153009    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.172842 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.172797    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:39:00.247769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.247675    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4gf\" (UniqueName: \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.247769 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.247729    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.247999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.247955    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrn5\" (UniqueName: \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.247999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.247990    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.248764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.248743    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.274325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.274291    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4gf\" (UniqueName: \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\") pod \"payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.335444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.335407    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:00.349387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.349353    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.349551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.349535    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrn5\" (UniqueName: \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.350300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.350276    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.385295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.383305    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrn5\" (UniqueName: \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.470181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.470137    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:00.823205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.823145    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:39:00.838445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.838415    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ca11fd-24f5-42f8-85ae-125c19ffaba5" path="/var/lib/kubelet/pods/08ca11fd-24f5-42f8-85ae-125c19ffaba5/volumes"
Oct 30 15:39:00.841423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.838858    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" event={"ID":"82fc936e-4a4c-4628-b1f7-4649dd55dadf","Type":"ContainerStarted","Data":"bd717a4e9578024ed6d9c12d83ab636b7ce9390127039da59752a7145090917f"}
Oct 30 15:39:00.841423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.838925    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:39:00.864152 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.864118    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:39:00.969704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:00.969673    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 15:39:01.011576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.011541    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 15:39:01.619963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.619930    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:39:01.634787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.634746    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.634996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.634926    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.634996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.634969    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635031    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635156    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635231    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635260    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635292    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635383    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:01.635476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635428    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\") pod \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\" (UID: \"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5\") "
Oct 30 15:39:01.635539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635492    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:01.635653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635628    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:01.635915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.635890    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:01.636190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.636169    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.636294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.636195    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.636294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.636222    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.636294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.636234    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.636628 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.636522    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:01.642151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.642118    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:01.643029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.642998    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:39:01.644692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.644659    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "kube-api-access-76fwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:01.645040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.645016    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token" pod "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" (UID: "ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:01.767894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.742652    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fwf\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-kube-api-access-76fwf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.767894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.742690    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.767894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.742705    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.767894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.742717    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.767894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.742729    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:01.891610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.890299    5064 generic.go:356] "Generic (PLEG): container finished" podID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerID="37e063e38ccde8d356b01e40567faffabe7ce8304b5807a69b0f2981bddc164f" exitCode=0
Oct 30 15:39:01.891610 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.890439    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" event={"ID":"193959a9-6f5a-4d29-a1c0-be5433a7861f","Type":"ContainerDied","Data":"37e063e38ccde8d356b01e40567faffabe7ce8304b5807a69b0f2981bddc164f"}
Oct 30 15:39:01.900387 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.900346    5064 generic.go:356] "Generic (PLEG): container finished" podID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerID="96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07" exitCode=0
Oct 30 15:39:01.900609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.900587    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerDied","Data":"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"}
Oct 30 15:39:01.900679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.900655    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg" event={"ID":"ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5","Type":"ContainerDied","Data":"00a8819f36bf48d217392799ff4743fe2bc5006eb1a4586ff361d4674c91a9d1"}
Oct 30 15:39:01.900739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.900687    5064 scope.go:122] "RemoveContainer" containerID="96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"
Oct 30 15:39:01.900876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.900861    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"
Oct 30 15:39:01.920109 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.919152    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" event={"ID":"3cb4f80a-42f1-4626-b736-cadd7c37453f","Type":"ContainerStarted","Data":"c1c2b2853debca84b465e851ac8024c8884fbf8cfce270eea644684aa374afc5"}
Oct 30 15:39:01.947092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.946973    5064 scope.go:122] "RemoveContainer" containerID="ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"
Oct 30 15:39:01.965002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.964820    5064 scope.go:122] "RemoveContainer" containerID="96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"
Oct 30 15:39:01.966031 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:01.965997    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07\": container with ID starting with 96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07 not found: ID does not exist" containerID="96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"
Oct 30 15:39:01.966162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.966057    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07"} err="failed to get container status \"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07\": rpc error: code = NotFound desc = could not find container \"96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07\": container with ID starting with 96bbc2c97ffa07b1602083c710c87b077e2dac7b66747b3e22855dd3cad51a07 not found: ID does not exist"
Oct 30 15:39:01.966162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.966086    5064 scope.go:122] "RemoveContainer" containerID="ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"
Oct 30 15:39:01.966638 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:01.966549    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3\": container with ID starting with ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3 not found: ID does not exist" containerID="ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"
Oct 30 15:39:01.966638 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.966589    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3"} err="failed to get container status \"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3\": rpc error: code = NotFound desc = could not find container \"ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3\": container with ID starting with ab692d8897239ddbe3b7929b68c01b549af770b9aa9417f9cf58ca9dfee8bce3 not found: ID does not exist"
Oct 30 15:39:01.991106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.991042    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:39:01.997633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:01.997565    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-stdrg"]
Oct 30 15:39:02.092370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.092292    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170035    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170100    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170138    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170206    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170285    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170327    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170379    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170456    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.171313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.170490    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\") pod \"193959a9-6f5a-4d29-a1c0-be5433a7861f\" (UID: \"193959a9-6f5a-4d29-a1c0-be5433a7861f\") "
Oct 30 15:39:02.173976 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.172547    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:02.175772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.175461    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.176198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.175963    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:02.176755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.176508    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:02.177310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.177248    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:02.178689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.178656    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:39:02.182606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.182554    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "kube-api-access-29rlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:02.182950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.182912    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:02.186172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.186143    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:02.190492 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.190456    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy" pod "193959a9-6f5a-4d29-a1c0-be5433a7861f" (UID: "193959a9-6f5a-4d29-a1c0-be5433a7861f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:02.276841 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276789    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.276841 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276845    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rlc\" (UniqueName: \"kubernetes.io/projected/193959a9-6f5a-4d29-a1c0-be5433a7861f-kube-api-access-29rlc\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276864    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276880    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276902    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276916    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193959a9-6f5a-4d29-a1c0-be5433a7861f-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276928    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.277102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.276941    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193959a9-6f5a-4d29-a1c0-be5433a7861f-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:02.728199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.728162    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" path="/var/lib/kubelet/pods/ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5/volumes"
Oct 30 15:39:02.931359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.931323    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"
Oct 30 15:39:02.931551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.931350    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb" event={"ID":"193959a9-6f5a-4d29-a1c0-be5433a7861f","Type":"ContainerDied","Data":"5171e4220791df067547a876656bfcbb1d3a9c0e7f7dbcdce1faea20ee1d9ee7"}
Oct 30 15:39:02.931551 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:02.931422    5064 scope.go:122] "RemoveContainer" containerID="37e063e38ccde8d356b01e40567faffabe7ce8304b5807a69b0f2981bddc164f"
Oct 30 15:39:03.022532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.022444    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:39:03.035260 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.035183    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-kktwb"]
Oct 30 15:39:03.943223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.943189    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" event={"ID":"3cb4f80a-42f1-4626-b736-cadd7c37453f","Type":"ContainerStarted","Data":"f1606117ea7b98d6565beb28680e7a8c0bbd6510e79c697b50490c85ae6cc3d0"}
Oct 30 15:39:03.943708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.943277    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:03.946700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.946665    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" event={"ID":"82fc936e-4a4c-4628-b1f7-4649dd55dadf","Type":"ContainerStarted","Data":"9be1490ba2b93220182e8ada535588200a29e08bad99b1b4cc5e9a10890bed69"}
Oct 30 15:39:03.946867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.946733    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:03.990474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:03.990411    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" podStartSLOduration=1.48866632 podStartE2EDuration="3.990395976s" totalImagesPullingTime="2.501729656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:03.986806827 +0000 UTC m=+3093.428697544" watchObservedRunningTime="2026-10-30 15:39:03.990395976 +0000 UTC m=+3093.432286693"
Oct 30 15:39:04.036760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:04.036695    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" podStartSLOduration=2.481298314 podStartE2EDuration="5.036675801s" totalImagesPullingTime="2.555377487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:04.032287784 +0000 UTC m=+3093.474178503" watchObservedRunningTime="2026-10-30 15:39:04.036675801 +0000 UTC m=+3093.478566520"
Oct 30 15:39:04.726167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:04.726132    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" path="/var/lib/kubelet/pods/193959a9-6f5a-4d29-a1c0-be5433a7861f/volumes"
Oct 30 15:39:05.971861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:05.971706    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing" containerID="cri-o://581b5fe06683251667dd9d7dd97dfc357ebb9a6888961b8999963d6bfcc3d5c5" gracePeriod=25
Oct 30 15:39:06.012651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:06.012584    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing" containerID="cri-o://d202c166d79afd62bded588c90157102a7dddc9d26d5bc3177fb2350d54e4f66" gracePeriod=25
Oct 30 15:39:07.394500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:07.394446    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:39:07.396021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:07.395271    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:39:09.492897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.492850    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:39:09.493397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.493120    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" podUID="728650fb-8a8b-439d-94d1-e8da3614699e" containerName="maas-api" containerID="cri-o://a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567" gracePeriod=30
Oct 30 15:39:09.848391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.848351    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:39:09.961561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.961527    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\") pod \"728650fb-8a8b-439d-94d1-e8da3614699e\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") "
Oct 30 15:39:09.961780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.961652    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\") pod \"728650fb-8a8b-439d-94d1-e8da3614699e\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") "
Oct 30 15:39:09.961780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.961753    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\" (UniqueName: \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\") pod \"728650fb-8a8b-439d-94d1-e8da3614699e\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") "
Oct 30 15:39:09.961942 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.961807    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\") pod \"728650fb-8a8b-439d-94d1-e8da3614699e\" (UID: \"728650fb-8a8b-439d-94d1-e8da3614699e\") "
Oct 30 15:39:09.962043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.962008    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca" pod "728650fb-8a8b-439d-94d1-e8da3614699e" (UID: "728650fb-8a8b-439d-94d1-e8da3614699e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:09.962556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.962531    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/728650fb-8a8b-439d-94d1-e8da3614699e-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:09.964314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.964289    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs" pod "728650fb-8a8b-439d-94d1-e8da3614699e" (UID: "728650fb-8a8b-439d-94d1-e8da3614699e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:39:09.964822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.964798    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48" pod "728650fb-8a8b-439d-94d1-e8da3614699e" (UID: "728650fb-8a8b-439d-94d1-e8da3614699e"). InnerVolumeSpecName "kube-api-access-k9m48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:09.966043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.966020    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls" pod "728650fb-8a8b-439d-94d1-e8da3614699e" (UID: "728650fb-8a8b-439d-94d1-e8da3614699e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:39:09.997570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.997532    5064 generic.go:356] "Generic (PLEG): container finished" podID="728650fb-8a8b-439d-94d1-e8da3614699e" containerID="a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567" exitCode=0
Oct 30 15:39:09.997776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.997589    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"
Oct 30 15:39:09.997776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.997621    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" event={"ID":"728650fb-8a8b-439d-94d1-e8da3614699e","Type":"ContainerDied","Data":"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"}
Oct 30 15:39:09.997776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.997688    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t" event={"ID":"728650fb-8a8b-439d-94d1-e8da3614699e","Type":"ContainerDied","Data":"2cd91264e3d16225866c825819dccf4438a3cdfca6632a286c7309aa2bb7f4df"}
Oct 30 15:39:09.997776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:09.997717    5064 scope.go:122] "RemoveContainer" containerID="a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"
Oct 30 15:39:10.034213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.034112    5064 scope.go:122] "RemoveContainer" containerID="a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"
Oct 30 15:39:10.046187 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:10.046026    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567\": container with ID starting with a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567 not found: ID does not exist" containerID="a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"
Oct 30 15:39:10.046187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.046082    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567"} err="failed to get container status \"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567\": rpc error: code = NotFound desc = could not find container \"a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567\": container with ID starting with a490809033fc9780e42400b23c1279344f5e53fc9e900d73b6741c9115027567 not found: ID does not exist"
Oct 30 15:39:10.068481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.067422    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:10.068481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.067464    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9m48\" (UniqueName: \"kubernetes.io/projected/728650fb-8a8b-439d-94d1-e8da3614699e-kube-api-access-k9m48\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:10.068481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.067481    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/728650fb-8a8b-439d-94d1-e8da3614699e-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:10.098609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.098560    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:39:10.139352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.139313    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-631d84-558468cddd-78z5t"]
Oct 30 15:39:10.727119 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:10.727086    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728650fb-8a8b-439d-94d1-e8da3614699e" path="/var/lib/kubelet/pods/728650fb-8a8b-439d-94d1-e8da3614699e/volumes"
Oct 30 15:39:14.957270 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:14.957228    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:39:14.957699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:14.957327    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:39:16.183075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:16.183032    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:39:16.183473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:16.183283    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy" containerID="cri-o://6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d" gracePeriod=30
Oct 30 15:39:17.376049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:17.376009    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:39:17.376557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:17.376009    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:39:19.435722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:19.435686    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:39:19.436254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:19.435930    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" containerID="cri-o://976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf" gracePeriod=30
Oct 30 15:39:21.258941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.258891    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:39:21.259413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.258967    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:39:21.498877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.498802    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:39:21.582881 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.582847    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.582953    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.582997    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583053    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583281 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583228    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583361 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583285    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583361 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583321    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583424    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:21.583474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583449    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583559    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\") pod \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\" (UID: \"d1dd2eeb-dd55-47b1-908b-1ded7a67517a\") "
Oct 30 15:39:21.583749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583725    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:21.583880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583768    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:21.583880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583658    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:21.584011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.583978    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:21.584267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.584248    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.584382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.584271    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.584382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.584284    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.584382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.584296    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.584382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.584308    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.600876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.594989    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:21.600876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.594996    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:39:21.600876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.595085    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:21.600876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.595122    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg" pod "d1dd2eeb-dd55-47b1-908b-1ded7a67517a" (UID: "d1dd2eeb-dd55-47b1-908b-1ded7a67517a"). InnerVolumeSpecName "kube-api-access-ksskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:21.686181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.686146    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.686181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.686177    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.686408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.686189    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksskg\" (UniqueName: \"kubernetes.io/projected/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-kube-api-access-ksskg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:21.686408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:21.686203    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dd2eeb-dd55-47b1-908b-1ded7a67517a-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:22.109081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.108967    5064 generic.go:356] "Generic (PLEG): container finished" podID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerID="6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d" exitCode=0
Oct 30 15:39:22.109081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.109033    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"
Oct 30 15:39:22.109328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.109066    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerDied","Data":"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"}
Oct 30 15:39:22.109328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.109156    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58" event={"ID":"d1dd2eeb-dd55-47b1-908b-1ded7a67517a","Type":"ContainerDied","Data":"a13dbd2c0372581ccd7b9ac1bd6fdd8a739aa36f531bd82b9576445ff8884a34"}
Oct 30 15:39:22.109328 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.109185    5064 scope.go:122] "RemoveContainer" containerID="6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"
Oct 30 15:39:22.125494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.125469    5064 scope.go:122] "RemoveContainer" containerID="f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"
Oct 30 15:39:22.136426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.136402    5064 scope.go:122] "RemoveContainer" containerID="6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"
Oct 30 15:39:22.136691 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:22.136670    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d\": container with ID starting with 6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d not found: ID does not exist" containerID="6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"
Oct 30 15:39:22.136783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.136704    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d"} err="failed to get container status \"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d\": rpc error: code = NotFound desc = could not find container \"6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d\": container with ID starting with 6432b3195f6eba46abf946b1d00836dfeb0aa716111dbaf2a8328344d1cd634d not found: ID does not exist"
Oct 30 15:39:22.136783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.136731    5064 scope.go:122] "RemoveContainer" containerID="f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"
Oct 30 15:39:22.137039 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:22.137005    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd\": container with ID starting with f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd not found: ID does not exist" containerID="f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"
Oct 30 15:39:22.137085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.137045    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd"} err="failed to get container status \"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd\": rpc error: code = NotFound desc = could not find container \"f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd\": container with ID starting with f51cb676016325d7a585ab6c3df741b610a6721444a4fc5f249ca1066060cfcd not found: ID does not exist"
Oct 30 15:39:22.180967 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.180931    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:39:22.184452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.184427    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-5gj58"]
Oct 30 15:39:22.723753 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:22.723720    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" path="/var/lib/kubelet/pods/d1dd2eeb-dd55-47b1-908b-1ded7a67517a/volumes"
Oct 30 15:39:24.743111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.743079    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:39:24.836606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836572    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.836805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836620    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.836805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836721    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.836805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836783    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836814    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836940    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837047 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.836976    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837034    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:24.837222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837054    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837222 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837098    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\") pod \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\" (UID: \"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f\") "
Oct 30 15:39:24.837373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837280    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:24.837456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837428    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:24.837688 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837665    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:24.837868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.837750    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.838052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.838033    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.838179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.838162    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.838287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.838096    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:24.839695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.839669    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:24.840045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.840011    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:39:24.840045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.840018    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "kube-api-access-rtsxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:24.841208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.841171    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy" pod "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" (UID: "1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:24.940186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940149    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.940186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940181    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.940186 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940191    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsxw\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-kube-api-access-rtsxw\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.940443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940200    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.940443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940209    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:24.940443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:24.940217    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:25.137112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.137073    5064 generic.go:356] "Generic (PLEG): container finished" podID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerID="976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf" exitCode=0
Oct 30 15:39:25.137339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.137131    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"
Oct 30 15:39:25.137339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.137149    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerDied","Data":"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"}
Oct 30 15:39:25.137339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.137205    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h" event={"ID":"1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f","Type":"ContainerDied","Data":"ef02703c2ecc91e64ea8d35f64935659d6829e56dd027c6231b3a94fc844e3a0"}
Oct 30 15:39:25.137339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.137236    5064 scope.go:122] "RemoveContainer" containerID="976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"
Oct 30 15:39:25.149467 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.149443    5064 scope.go:122] "RemoveContainer" containerID="94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"
Oct 30 15:39:25.173651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.173627    5064 scope.go:122] "RemoveContainer" containerID="976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"
Oct 30 15:39:25.174025 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:25.173963    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf\": container with ID starting with 976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf not found: ID does not exist" containerID="976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"
Oct 30 15:39:25.174025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.173997    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf"} err="failed to get container status \"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf\": rpc error: code = NotFound desc = could not find container \"976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf\": container with ID starting with 976f06efa1c7a031c252946dcf91a6d667cccdb3bf20ed9a393a91c22f000edf not found: ID does not exist"
Oct 30 15:39:25.174025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.174016    5064 scope.go:122] "RemoveContainer" containerID="94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"
Oct 30 15:39:25.174313 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:25.174288    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477\": container with ID starting with 94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477 not found: ID does not exist" containerID="94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"
Oct 30 15:39:25.174398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.174318    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477"} err="failed to get container status \"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477\": rpc error: code = NotFound desc = could not find container \"94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477\": container with ID starting with 94d658347b501b5b2b844990c9fd01c3d1c7eff87d221b763207a8a085db8477 not found: ID does not exist"
Oct 30 15:39:25.196977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.196939    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:39:25.210708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.210678    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-sqp5h"]
Oct 30 15:39:25.530657 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.530561    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:39:25.530910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.530866    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" podUID="27da83b4-c7c3-4336-8a33-03504738c856" containerName="maas-api" containerID="cri-o://a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48" gracePeriod=30
Oct 30 15:39:25.838972 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.838942    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:39:25.957861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.957780    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\") pod \"27da83b4-c7c3-4336-8a33-03504738c856\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") "
Oct 30 15:39:25.958169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.958141    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\") pod \"27da83b4-c7c3-4336-8a33-03504738c856\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") "
Oct 30 15:39:25.958277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.958255    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca" pod "27da83b4-c7c3-4336-8a33-03504738c856" (UID: "27da83b4-c7c3-4336-8a33-03504738c856"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:25.958344 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.958260    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\" (UniqueName: \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\") pod \"27da83b4-c7c3-4336-8a33-03504738c856\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") "
Oct 30 15:39:25.958393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.958345    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") pod \"27da83b4-c7c3-4336-8a33-03504738c856\" (UID: \"27da83b4-c7c3-4336-8a33-03504738c856\") "
Oct 30 15:39:25.959068 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.959039    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da83b4-c7c3-4336-8a33-03504738c856-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:25.960676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.960645    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls" pod "27da83b4-c7c3-4336-8a33-03504738c856" (UID: "27da83b4-c7c3-4336-8a33-03504738c856"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:39:25.960796 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.960696    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4" pod "27da83b4-c7c3-4336-8a33-03504738c856" (UID: "27da83b4-c7c3-4336-8a33-03504738c856"). InnerVolumeSpecName "kube-api-access-zgvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:25.960864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:25.960840    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs" pod "27da83b4-c7c3-4336-8a33-03504738c856" (UID: "27da83b4-c7c3-4336-8a33-03504738c856"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:39:26.061198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.061059    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvt4\" (UniqueName: \"kubernetes.io/projected/27da83b4-c7c3-4336-8a33-03504738c856-kube-api-access-zgvt4\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.061198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.061103    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.061198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.061119    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da83b4-c7c3-4336-8a33-03504738c856-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.181954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.181908    5064 generic.go:356] "Generic (PLEG): container finished" podID="27da83b4-c7c3-4336-8a33-03504738c856" containerID="a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48" exitCode=0
Oct 30 15:39:26.182174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.182095    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" event={"ID":"27da83b4-c7c3-4336-8a33-03504738c856","Type":"ContainerDied","Data":"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"}
Oct 30 15:39:26.182174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.182149    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz" event={"ID":"27da83b4-c7c3-4336-8a33-03504738c856","Type":"ContainerDied","Data":"ca975707ed5a30d3fda8c1b1ddaa4a7a6b3248c687a65bb382f49e21fe01e096"}
Oct 30 15:39:26.182303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.182183    5064 scope.go:122] "RemoveContainer" containerID="a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"
Oct 30 15:39:26.182360 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.182307    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"
Oct 30 15:39:26.210954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.206249    5064 generic.go:356] "Generic (PLEG): container finished" podID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerID="581b5fe06683251667dd9d7dd97dfc357ebb9a6888961b8999963d6bfcc3d5c5" exitCode=0
Oct 30 15:39:26.210954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.206459    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" event={"ID":"d274a787-7d52-46c4-ab60-e1db2d53b04f","Type":"ContainerDied","Data":"581b5fe06683251667dd9d7dd97dfc357ebb9a6888961b8999963d6bfcc3d5c5"}
Oct 30 15:39:26.210954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.208568    5064 generic.go:356] "Generic (PLEG): container finished" podID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerID="d202c166d79afd62bded588c90157102a7dddc9d26d5bc3177fb2350d54e4f66" exitCode=0
Oct 30 15:39:26.210954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.208710    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" event={"ID":"d2b507ad-1592-40a9-88e8-cf4ecb3a6698","Type":"ContainerDied","Data":"d202c166d79afd62bded588c90157102a7dddc9d26d5bc3177fb2350d54e4f66"}
Oct 30 15:39:26.229315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.227281    5064 scope.go:122] "RemoveContainer" containerID="a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"
Oct 30 15:39:26.229315 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:26.228851    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48\": container with ID starting with a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48 not found: ID does not exist" containerID="a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"
Oct 30 15:39:26.229315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.228882    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48"} err="failed to get container status \"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48\": rpc error: code = NotFound desc = could not find container \"a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48\": container with ID starting with a96c898bcde09ac11bf4d18c7ab329ffd597b3cb9338ce411e87339ba8e5bb48 not found: ID does not exist"
Oct 30 15:39:26.253371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.253334    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:39:26.259002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.258890    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e8bf0-7685b4859f-q54dz"]
Oct 30 15:39:26.383690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.383656    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 15:39:26.418719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.418685    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 15:39:26.468619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.468584    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\" (UniqueName: \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\") pod \"d274a787-7d52-46c4-ab60-e1db2d53b04f\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") "
Oct 30 15:39:26.468821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.468676    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\") pod \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") "
Oct 30 15:39:26.468821 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.468755    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\") pod \"d274a787-7d52-46c4-ab60-e1db2d53b04f\" (UID: \"d274a787-7d52-46c4-ab60-e1db2d53b04f\") "
Oct 30 15:39:26.468983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.468911    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\" (UniqueName: \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\") pod \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\" (UID: \"d2b507ad-1592-40a9-88e8-cf4ecb3a6698\") "
Oct 30 15:39:26.469176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.469153    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume" pod "d2b507ad-1592-40a9-88e8-cf4ecb3a6698" (UID: "d2b507ad-1592-40a9-88e8-cf4ecb3a6698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:26.469249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.469180    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume" pod "d274a787-7d52-46c4-ab60-e1db2d53b04f" (UID: "d274a787-7d52-46c4-ab60-e1db2d53b04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:26.469625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.469603    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.469625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.469627    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d274a787-7d52-46c4-ab60-e1db2d53b04f-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.471642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.471616    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc" pod "d2b507ad-1592-40a9-88e8-cf4ecb3a6698" (UID: "d2b507ad-1592-40a9-88e8-cf4ecb3a6698"). InnerVolumeSpecName "kube-api-access-b5vfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:26.472045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.472024    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc" pod "d274a787-7d52-46c4-ab60-e1db2d53b04f" (UID: "d274a787-7d52-46c4-ab60-e1db2d53b04f"). InnerVolumeSpecName "kube-api-access-gbqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:26.592923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.592628    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqgc\" (UniqueName: \"kubernetes.io/projected/d274a787-7d52-46c4-ab60-e1db2d53b04f-kube-api-access-gbqgc\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.592923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.592676    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vfc\" (UniqueName: \"kubernetes.io/projected/d2b507ad-1592-40a9-88e8-cf4ecb3a6698-kube-api-access-b5vfc\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:26.725309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.725274    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" path="/var/lib/kubelet/pods/1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f/volumes"
Oct 30 15:39:26.725813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:26.725799    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27da83b4-c7c3-4336-8a33-03504738c856" path="/var/lib/kubelet/pods/27da83b4-c7c3-4336-8a33-03504738c856/volumes"
Oct 30 15:39:27.221170 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.221125    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9" event={"ID":"d274a787-7d52-46c4-ab60-e1db2d53b04f","Type":"ContainerDied","Data":"8d51242db7a5d3b1df334a98c5c0ba93f52adb8380b717a5af195ba47de421db"}
Oct 30 15:39:27.221675 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.221219    5064 scope.go:122] "RemoveContainer" containerID="581b5fe06683251667dd9d7dd97dfc357ebb9a6888961b8999963d6bfcc3d5c5"
Oct 30 15:39:27.221675 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.221147    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"
Oct 30 15:39:27.224163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.223949    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q" event={"ID":"d2b507ad-1592-40a9-88e8-cf4ecb3a6698","Type":"ContainerDied","Data":"eac4276ce24db6cdc11d72974852c3193d5d1540f8cabc40c485d96548753883"}
Oct 30 15:39:27.224163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.224046    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"
Oct 30 15:39:27.244198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.244170    5064 scope.go:122] "RemoveContainer" containerID="d202c166d79afd62bded588c90157102a7dddc9d26d5bc3177fb2350d54e4f66"
Oct 30 15:39:27.280473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.280436    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 15:39:27.293162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.293116    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-05d396-6dbb6b6496-xhkd9"]
Oct 30 15:39:27.328216 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.328174    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 15:39:27.332327 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:27.332292    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-05d396-67747b94ff-brj9q"]
Oct 30 15:39:28.725439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:28.725403    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" path="/var/lib/kubelet/pods/d274a787-7d52-46c4-ab60-e1db2d53b04f/volumes"
Oct 30 15:39:28.726094 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:28.726076    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" path="/var/lib/kubelet/pods/d2b507ad-1592-40a9-88e8-cf4ecb3a6698/volumes"
Oct 30 15:39:29.632811 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:29.632776    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:39:33.128372 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:33.128336    5064 scope.go:122] "RemoveContainer" containerID="b756a80b856c4905dc1df27d20fb26b7f14c4de42ab00f39c25af175bb9fd58a"
Oct 30 15:39:46.021817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:46.021778    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:39:46.023029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:46.022040    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" containerID="cri-o://7962a934896575b5aca27d4b6744e4287cf39a33e1c807aa94e078abd4b78630" gracePeriod=30
Oct 30 15:39:50.152244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:50.152206    5064 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:39:50.152633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:50.152263    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:39:51.452962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.452917    5064 generic.go:356] "Generic (PLEG): container finished" podID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerID="7962a934896575b5aca27d4b6744e4287cf39a33e1c807aa94e078abd4b78630" exitCode=0
Oct 30 15:39:51.453416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.452985    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerDied","Data":"7962a934896575b5aca27d4b6744e4287cf39a33e1c807aa94e078abd4b78630"}
Oct 30 15:39:51.453416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.453061    5064 scope.go:122] "RemoveContainer" containerID="8917048906aff05ef01bea6321a91643342a12ec82f5b5031cce20dd00695db2"
Oct 30 15:39:51.730773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.730739    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:39:51.867658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867622    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.867887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867688    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.867887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867726    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.867887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867762    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.867887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867875    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.868139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.867989    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.868139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868018    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.868139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868116    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.868307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868111    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:51.868307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868155    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\") pod \"69852b89-3caa-4c13-a40a-8bb194876a0e\" (UID: \"69852b89-3caa-4c13-a40a-8bb194876a0e\") "
Oct 30 15:39:51.868616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868518    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:51.868616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868534    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:51.868616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868557    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:39:51.868616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.868589    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:51.869200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.869175    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.869280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.869205    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.869280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.869221    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.869280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.869238    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69852b89-3caa-4c13-a40a-8bb194876a0e-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.869280 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.869252    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.870423 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.870401    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:39:51.870547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.870526    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "kube-api-access-n2b9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:51.870602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.870555    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:39:51.870707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.870691    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token" pod "69852b89-3caa-4c13-a40a-8bb194876a0e" (UID: "69852b89-3caa-4c13-a40a-8bb194876a0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:39:51.970495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.970403    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.970495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.970437    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.970495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.970451    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2b9z\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-kube-api-access-n2b9z\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:51.970495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:51.970463    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69852b89-3caa-4c13-a40a-8bb194876a0e-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:39:52.461525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.461487    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"
Oct 30 15:39:52.461975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.461525    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt" event={"ID":"69852b89-3caa-4c13-a40a-8bb194876a0e","Type":"ContainerDied","Data":"62b800060130b73e6f30fc5970d978d66e80f4a5a27502aa053e1fac9eaf87a4"}
Oct 30 15:39:52.461975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.461589    5064 scope.go:122] "RemoveContainer" containerID="7962a934896575b5aca27d4b6744e4287cf39a33e1c807aa94e078abd4b78630"
Oct 30 15:39:52.542028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.541993    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:39:52.549319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.549280    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-hcnrt"]
Oct 30 15:39:52.724191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:39:52.724105    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" path="/var/lib/kubelet/pods/69852b89-3caa-4c13-a40a-8bb194876a0e/volumes"
Oct 30 15:39:59.650427 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:39:59.650380    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:40:07.218024 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.217910    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 15:40:07.218534 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.218176    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerName="maas-api" containerID="cri-o://02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b" gracePeriod=30
Oct 30 15:40:07.566137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.566109    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 15:40:07.607244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607163    5064 generic.go:356] "Generic (PLEG): container finished" podID="30b59509-9b9a-4399-9f13-4453158404e4" containerID="f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe" exitCode=0
Oct 30 15:40:07.607399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607238    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerDied","Data":"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"}
Oct 30 15:40:07.607732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607705    5064 scope.go:122] "RemoveContainer" containerID="f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"
Oct 30 15:40:07.607950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607771    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\" (UniqueName: \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\") pod \"bd778804-48d0-4ef0-9b62-683b1ba8f171\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") "
Oct 30 15:40:07.607950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607820    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\") pod \"bd778804-48d0-4ef0-9b62-683b1ba8f171\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") "
Oct 30 15:40:07.607950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607943    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\") pod \"bd778804-48d0-4ef0-9b62-683b1ba8f171\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") "
Oct 30 15:40:07.608274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.607981    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") pod \"bd778804-48d0-4ef0-9b62-683b1ba8f171\" (UID: \"bd778804-48d0-4ef0-9b62-683b1ba8f171\") "
Oct 30 15:40:07.608918 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.608525    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca" pod "bd778804-48d0-4ef0-9b62-683b1ba8f171" (UID: "bd778804-48d0-4ef0-9b62-683b1ba8f171"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609005    5064 generic.go:356] "Generic (PLEG): container finished" podID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerID="02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b" exitCode=0
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609033    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" event={"ID":"bd778804-48d0-4ef0-9b62-683b1ba8f171","Type":"ContainerDied","Data":"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"}
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609065    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609087    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" event={"ID":"bd778804-48d0-4ef0-9b62-683b1ba8f171","Type":"ContainerDied","Data":"c9b3a6ea8eee68110a8fd9eb6cafd01fcae77fb68820fdb69907f12b99c00ccd"}
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609115    5064 scope.go:122] "RemoveContainer" containerID="02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"
Oct 30 15:40:07.609336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609268    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd778804-48d0-4ef0-9b62-683b1ba8f171-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:07.609886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609725    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:40:07.609886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609847    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:40:07.610016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.609904    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:40:07.613335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.610547    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs" pod "bd778804-48d0-4ef0-9b62-683b1ba8f171" (UID: "bd778804-48d0-4ef0-9b62-683b1ba8f171"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:07.613335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.610657    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls" pod "bd778804-48d0-4ef0-9b62-683b1ba8f171" (UID: "bd778804-48d0-4ef0-9b62-683b1ba8f171"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:07.613335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.610764    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj" pod "bd778804-48d0-4ef0-9b62-683b1ba8f171" (UID: "bd778804-48d0-4ef0-9b62-683b1ba8f171"). InnerVolumeSpecName "kube-api-access-jbmsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:07.744856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.744176    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmsj\" (UniqueName: \"kubernetes.io/projected/bd778804-48d0-4ef0-9b62-683b1ba8f171-kube-api-access-jbmsj\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:07.744856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.744208    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:07.744856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.744223    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd778804-48d0-4ef0-9b62-683b1ba8f171-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:07.749097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.747374    5064 scope.go:122] "RemoveContainer" containerID="02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"
Oct 30 15:40:07.749097 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:07.748070    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b\": container with ID starting with 02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b not found: ID does not exist" containerID="02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"
Oct 30 15:40:07.749097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.748114    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b"} err="failed to get container status \"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b\": rpc error: code = NotFound desc = could not find container \"02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b\": container with ID starting with 02e4c1f06279edc8304f836a91bf4ba2f84e65fd41a1e939015c75ef53405e8b not found: ID does not exist"
Oct 30 15:40:07.962232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.962192    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 15:40:07.968757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:07.968721    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng"]
Oct 30 15:40:08.649908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:08.649824    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerStarted","Data":"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"}
Oct 30 15:40:08.745982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:08.745942    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" path="/var/lib/kubelet/pods/bd778804-48d0-4ef0-9b62-683b1ba8f171/volumes"
Oct 30 15:40:09.385058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:09.385023    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:40:09.386300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:09.386272    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 15:40:09.386448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:09.386328    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 15:40:10.385496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:10.385459    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 15:40:10.385955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:10.385529    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 15:40:10.434855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:10.430981    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-05d396-7988b94f4f-p4zng" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:40:11.385934 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:11.385894    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 15:40:11.386416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:11.385971    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 15:40:11.658443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:11.658359    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:40:11.658604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:11.658563    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" podUID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" containerName="maas-api" containerID="cri-o://814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed" gracePeriod=30
Oct 30 15:40:11.693529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:11.693491    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:40:12.101590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.101557    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:40:12.203706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.203202    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\") pod \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") "
Oct 30 15:40:12.203706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.203297    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\") pod \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") "
Oct 30 15:40:12.203706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.203380    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\") pod \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") "
Oct 30 15:40:12.203706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.203642    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca" pod "1a78a95e-8f90-4712-ab7a-f93ee42fd54f" (UID: "1a78a95e-8f90-4712-ab7a-f93ee42fd54f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:12.204277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.204139    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\" (UniqueName: \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\") pod \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\" (UID: \"1a78a95e-8f90-4712-ab7a-f93ee42fd54f\") "
Oct 30 15:40:12.205003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.204959    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:12.205804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.205775    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls" pod "1a78a95e-8f90-4712-ab7a-f93ee42fd54f" (UID: "1a78a95e-8f90-4712-ab7a-f93ee42fd54f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:12.206600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.206577    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld" pod "1a78a95e-8f90-4712-ab7a-f93ee42fd54f" (UID: "1a78a95e-8f90-4712-ab7a-f93ee42fd54f"). InnerVolumeSpecName "kube-api-access-fzsld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:12.206870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.206814    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs" pod "1a78a95e-8f90-4712-ab7a-f93ee42fd54f" (UID: "1a78a95e-8f90-4712-ab7a-f93ee42fd54f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:12.345908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.306943    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:12.345908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.306976    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:12.345908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.306992    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzsld\" (UniqueName: \"kubernetes.io/projected/1a78a95e-8f90-4712-ab7a-f93ee42fd54f-kube-api-access-fzsld\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:12.417065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.416704    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 15:40:12.418884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.418442    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 15:40:12.703863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.700962    5064 generic.go:356] "Generic (PLEG): container finished" podID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" containerID="814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed" exitCode=0
Oct 30 15:40:12.703863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.701076    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" event={"ID":"1a78a95e-8f90-4712-ab7a-f93ee42fd54f","Type":"ContainerDied","Data":"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"}
Oct 30 15:40:12.703863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.701131    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr" event={"ID":"1a78a95e-8f90-4712-ab7a-f93ee42fd54f","Type":"ContainerDied","Data":"af6b168d6327953c8a0a2554b50981a79b742947914804fd4b3c079662ec5fbf"}
Oct 30 15:40:12.703863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.701162    5064 scope.go:122] "RemoveContainer" containerID="814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"
Oct 30 15:40:12.703863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.701314    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"
Oct 30 15:40:12.727852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.724004    5064 scope.go:122] "RemoveContainer" containerID="814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"
Oct 30 15:40:12.733852 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:12.733263    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed\": container with ID starting with 814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed not found: ID does not exist" containerID="814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"
Oct 30 15:40:12.733852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.733308    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed"} err="failed to get container status \"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed\": rpc error: code = NotFound desc = could not find container \"814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed\": container with ID starting with 814c558355b5c06dfcc461c0f3b6883b62f8f9ddcb3ac2d6cb8fa041ab0bcfed not found: ID does not exist"
Oct 30 15:40:12.771415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.771371    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:40:12.784481 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:12.784443    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e30f62-7b89cc7496-np6xr"]
Oct 30 15:40:13.385478 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:13.385435    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 15:40:13.385678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:13.385500    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 15:40:14.162264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.162225    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 15:40:14.390339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.390303    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:40:14.391545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.391518    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:40:14.551533 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.551459    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:40:14.552211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.551908    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerName="payload-processing" containerID="cri-o://9be1490ba2b93220182e8ada535588200a29e08bad99b1b4cc5e9a10890bed69" gracePeriod=30
Oct 30 15:40:14.574742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.574710    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:40:14.579161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.579083    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerName="payload-pre-processing" containerID="cri-o://f1606117ea7b98d6565beb28680e7a8c0bbd6510e79c697b50490c85ae6cc3d0" gracePeriod=30
Oct 30 15:40:14.723764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.720921    5064 generic.go:356] "Generic (PLEG): container finished" podID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerID="9be1490ba2b93220182e8ada535588200a29e08bad99b1b4cc5e9a10890bed69" exitCode=0
Oct 30 15:40:14.723764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.723144    5064 generic.go:356] "Generic (PLEG): container finished" podID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerID="f1606117ea7b98d6565beb28680e7a8c0bbd6510e79c697b50490c85ae6cc3d0" exitCode=0
Oct 30 15:40:14.727941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.727344    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" path="/var/lib/kubelet/pods/1a78a95e-8f90-4712-ab7a-f93ee42fd54f/volumes"
Oct 30 15:40:14.727941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.727693    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" event={"ID":"82fc936e-4a4c-4628-b1f7-4649dd55dadf","Type":"ContainerDied","Data":"9be1490ba2b93220182e8ada535588200a29e08bad99b1b4cc5e9a10890bed69"}
Oct 30 15:40:14.727941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:14.727734    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" event={"ID":"3cb4f80a-42f1-4626-b736-cadd7c37453f","Type":"ContainerDied","Data":"f1606117ea7b98d6565beb28680e7a8c0bbd6510e79c697b50490c85ae6cc3d0"}
Oct 30 15:40:15.017752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.017398    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:40:15.109924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.092738    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\" (UniqueName: \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\") pod \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") "
Oct 30 15:40:15.109924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.092851    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\") pod \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\" (UID: \"82fc936e-4a4c-4628-b1f7-4649dd55dadf\") "
Oct 30 15:40:15.109924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.094881    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume" pod "82fc936e-4a4c-4628-b1f7-4649dd55dadf" (UID: "82fc936e-4a4c-4628-b1f7-4649dd55dadf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:15.111977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.111941    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf" pod "82fc936e-4a4c-4628-b1f7-4649dd55dadf" (UID: "82fc936e-4a4c-4628-b1f7-4649dd55dadf"). InnerVolumeSpecName "kube-api-access-pv4gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:15.128190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.128151    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:40:15.195954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.195914    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\" (UniqueName: \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\") pod \"3cb4f80a-42f1-4626-b736-cadd7c37453f\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") "
Oct 30 15:40:15.196468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.196209    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\") pod \"3cb4f80a-42f1-4626-b736-cadd7c37453f\" (UID: \"3cb4f80a-42f1-4626-b736-cadd7c37453f\") "
Oct 30 15:40:15.196865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.196820    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4gf\" (UniqueName: \"kubernetes.io/projected/82fc936e-4a4c-4628-b1f7-4649dd55dadf-kube-api-access-pv4gf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:15.196996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.196871    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fc936e-4a4c-4628-b1f7-4649dd55dadf-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:15.197654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.197622    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume" pod "3cb4f80a-42f1-4626-b736-cadd7c37453f" (UID: "3cb4f80a-42f1-4626-b736-cadd7c37453f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:15.203500 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.203424    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5" pod "3cb4f80a-42f1-4626-b736-cadd7c37453f" (UID: "3cb4f80a-42f1-4626-b736-cadd7c37453f"). InnerVolumeSpecName "kube-api-access-7xrn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:15.298263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.298227    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb4f80a-42f1-4626-b736-cadd7c37453f-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:15.298263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.298260    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrn5\" (UniqueName: \"kubernetes.io/projected/3cb4f80a-42f1-4626-b736-cadd7c37453f-kube-api-access-7xrn5\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:15.731808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.731780    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"
Oct 30 15:40:15.732043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.731818    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" event={"ID":"3cb4f80a-42f1-4626-b736-cadd7c37453f","Type":"ContainerDied","Data":"c1c2b2853debca84b465e851ac8024c8884fbf8cfce270eea644684aa374afc5"}
Oct 30 15:40:15.732043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.731920    5064 scope.go:122] "RemoveContainer" containerID="f1606117ea7b98d6565beb28680e7a8c0bbd6510e79c697b50490c85ae6cc3d0"
Oct 30 15:40:15.733550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.733524    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" event={"ID":"82fc936e-4a4c-4628-b1f7-4649dd55dadf","Type":"ContainerDied","Data":"bd717a4e9578024ed6d9c12d83ab636b7ce9390127039da59752a7145090917f"}
Oct 30 15:40:15.733696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.733571    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"
Oct 30 15:40:15.743458 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.743439    5064 scope.go:122] "RemoveContainer" containerID="9be1490ba2b93220182e8ada535588200a29e08bad99b1b4cc5e9a10890bed69"
Oct 30 15:40:15.790223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.790177    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:40:15.796325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.796293    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh"]
Oct 30 15:40:15.832564 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.832482    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:40:15.838608 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.838578    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd"]
Oct 30 15:40:15.962406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.962357    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-6988cd4f79-pwhxh" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.196:9004: i/o timeout"
Oct 30 15:40:15.962406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:15.962357    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-755478fb84-qbjwd" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.195:9004: i/o timeout"
Oct 30 15:40:16.761531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:16.761484    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" path="/var/lib/kubelet/pods/3cb4f80a-42f1-4626-b736-cadd7c37453f/volumes"
Oct 30 15:40:16.762351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:16.762330    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" path="/var/lib/kubelet/pods/82fc936e-4a4c-4628-b1f7-4649dd55dadf/volumes"
Oct 30 15:40:18.829612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:18.829581    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:40:19.048139 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.048102    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:40:19.048353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.048330    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" podUID="89094cc8-092a-4281-ace4-52ef6af5718c" containerName="maas-api" containerID="cri-o://ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9" gracePeriod=30
Oct 30 15:40:19.050803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.050782    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:40:19.162605 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.162522    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" podUID="6afd3f97-5981-4624-91b5-519fa4b031a8" containerName="payload-pre-processing" containerID="cri-o://0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612" gracePeriod=25
Oct 30 15:40:19.413468 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.413018    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:40:19.461223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.460023    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\" (UniqueName: \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\") pod \"89094cc8-092a-4281-ace4-52ef6af5718c\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") "
Oct 30 15:40:19.461223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.460075    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\") pod \"89094cc8-092a-4281-ace4-52ef6af5718c\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") "
Oct 30 15:40:19.461223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.460154    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\") pod \"89094cc8-092a-4281-ace4-52ef6af5718c\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") "
Oct 30 15:40:19.461223 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.460287    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\") pod \"89094cc8-092a-4281-ace4-52ef6af5718c\" (UID: \"89094cc8-092a-4281-ace4-52ef6af5718c\") "
Oct 30 15:40:19.462654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.462600    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca" pod "89094cc8-092a-4281-ace4-52ef6af5718c" (UID: "89094cc8-092a-4281-ace4-52ef6af5718c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:19.464320 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.464283    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs" pod "89094cc8-092a-4281-ace4-52ef6af5718c" (UID: "89094cc8-092a-4281-ace4-52ef6af5718c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:19.486367 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.486067    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh" pod "89094cc8-092a-4281-ace4-52ef6af5718c" (UID: "89094cc8-092a-4281-ace4-52ef6af5718c"). InnerVolumeSpecName "kube-api-access-6wpgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:19.511052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.510988    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls" pod "89094cc8-092a-4281-ace4-52ef6af5718c" (UID: "89094cc8-092a-4281-ace4-52ef6af5718c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:19.576295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.574980    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpgh\" (UniqueName: \"kubernetes.io/projected/89094cc8-092a-4281-ace4-52ef6af5718c-kube-api-access-6wpgh\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.576295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.575019    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.576295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.575033    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89094cc8-092a-4281-ace4-52ef6af5718c-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.576295 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.575047    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89094cc8-092a-4281-ace4-52ef6af5718c-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.737986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.737956    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 15:40:19.777394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.777350    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\") pod \"6afd3f97-5981-4624-91b5-519fa4b031a8\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") "
Oct 30 15:40:19.777604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.777470    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\" (UniqueName: \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\") pod \"6afd3f97-5981-4624-91b5-519fa4b031a8\" (UID: \"6afd3f97-5981-4624-91b5-519fa4b031a8\") "
Oct 30 15:40:19.777849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.777808    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume" pod "6afd3f97-5981-4624-91b5-519fa4b031a8" (UID: "6afd3f97-5981-4624-91b5-519fa4b031a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:19.778050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.778031    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd3f97-5981-4624-91b5-519fa4b031a8-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.779815 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.779788    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw" pod "6afd3f97-5981-4624-91b5-519fa4b031a8" (UID: "6afd3f97-5981-4624-91b5-519fa4b031a8"). InnerVolumeSpecName "kube-api-access-8mltw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:19.788443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.788408    5064 generic.go:356] "Generic (PLEG): container finished" podID="89094cc8-092a-4281-ace4-52ef6af5718c" containerID="ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9" exitCode=0
Oct 30 15:40:19.788613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.788450    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"
Oct 30 15:40:19.788613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.788486    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" event={"ID":"89094cc8-092a-4281-ace4-52ef6af5718c","Type":"ContainerDied","Data":"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"}
Oct 30 15:40:19.788613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.788543    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j" event={"ID":"89094cc8-092a-4281-ace4-52ef6af5718c","Type":"ContainerDied","Data":"3ef9d582b6d6bb5f0cd8a49d0c30889611297bdc71bc5902360c106ecb235f0c"}
Oct 30 15:40:19.788613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.788572    5064 scope.go:122] "RemoveContainer" containerID="ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"
Oct 30 15:40:19.790242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.790215    5064 generic.go:356] "Generic (PLEG): container finished" podID="6afd3f97-5981-4624-91b5-519fa4b031a8" containerID="0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612" exitCode=0
Oct 30 15:40:19.790389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.790263    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"
Oct 30 15:40:19.790389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.790290    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" event={"ID":"6afd3f97-5981-4624-91b5-519fa4b031a8","Type":"ContainerDied","Data":"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"}
Oct 30 15:40:19.790389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.790330    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv" event={"ID":"6afd3f97-5981-4624-91b5-519fa4b031a8","Type":"ContainerDied","Data":"77654b67d7170278a1e7ac8a9e7db2bdac5afadd25ffae7d0271bae885b0bd71"}
Oct 30 15:40:19.802415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.802330    5064 scope.go:122] "RemoveContainer" containerID="ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"
Oct 30 15:40:19.802778 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:19.802739    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9\": container with ID starting with ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9 not found: ID does not exist" containerID="ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"
Oct 30 15:40:19.802968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.802777    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9"} err="failed to get container status \"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9\": rpc error: code = NotFound desc = could not find container \"ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9\": container with ID starting with ceb895b47b19b8619e0f6636b648e982ffbca375d536bdb92e473e7a051367e9 not found: ID does not exist"
Oct 30 15:40:19.802968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.802804    5064 scope.go:122] "RemoveContainer" containerID="0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"
Oct 30 15:40:19.817639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.817504    5064 scope.go:122] "RemoveContainer" containerID="0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"
Oct 30 15:40:19.818609 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:19.818498    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612\": container with ID starting with 0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612 not found: ID does not exist" containerID="0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"
Oct 30 15:40:19.818609 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.818537    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612"} err="failed to get container status \"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612\": rpc error: code = NotFound desc = could not find container \"0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612\": container with ID starting with 0f8a7a9f6a92408780d4a20f884360c671d7e7baf105a2e2f0b69ba8d9b13612 not found: ID does not exist"
Oct 30 15:40:19.859642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.859598    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:40:19.865493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.865441    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-76473c-55fb6887d9-ssq5j"]
Oct 30 15:40:19.880849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.880800    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mltw\" (UniqueName: \"kubernetes.io/projected/6afd3f97-5981-4624-91b5-519fa4b031a8-kube-api-access-8mltw\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:19.950884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.948574    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 15:40:19.953017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:19.952984    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2c8d89-768b75769f-vhxdv"]
Oct 30 15:40:20.723538 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:20.723506    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afd3f97-5981-4624-91b5-519fa4b031a8" path="/var/lib/kubelet/pods/6afd3f97-5981-4624-91b5-519fa4b031a8/volumes"
Oct 30 15:40:20.724029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:20.724015    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89094cc8-092a-4281-ace4-52ef6af5718c" path="/var/lib/kubelet/pods/89094cc8-092a-4281-ace4-52ef6af5718c/volumes"
Oct 30 15:40:23.233350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:23.233312    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:40:23.233874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:23.233582    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" containerID="cri-o://e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b" gracePeriod=30
Oct 30 15:40:26.420950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:26.420908    5064 patch_prober.go:29] interesting pod/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:40:26.421358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:26.421003    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:40:27.494382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:27.494347    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:40:27.494863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:27.494605    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy" containerID="cri-o://479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406" gracePeriod=30
Oct 30 15:40:28.516889 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.516811    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:40:28.647583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647542    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.647803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647654    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.647803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647691    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.647803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647745    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.647803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647783    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.648061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647813    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.648061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647907    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.648061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.647917    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:28.648229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648058    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.648229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648061    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:28.648229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648146    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:28.648229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648148    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\") pod \"7176ec41-5592-4f76-b284-203843247cf8\" (UID: \"7176ec41-5592-4f76-b284-203843247cf8\") "
Oct 30 15:40:28.648438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648256    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:28.648438 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648394    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:28.648882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648855    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.648882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648882    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7176ec41-5592-4f76-b284-203843247cf8-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.649084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648891    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.649084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648899    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.649084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.648907    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.650411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.650387    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:28.650411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.650401    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "kube-api-access-45nqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:28.650590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.650554    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:40:28.650661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.650653    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy" pod "7176ec41-5592-4f76-b284-203843247cf8" (UID: "7176ec41-5592-4f76-b284-203843247cf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:28.750784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.750747    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45nqg\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-kube-api-access-45nqg\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.750784 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.750781    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7176ec41-5592-4f76-b284-203843247cf8-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.751039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.750795    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7176ec41-5592-4f76-b284-203843247cf8-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.751039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.750807    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7176ec41-5592-4f76-b284-203843247cf8-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:28.857161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.857073    5064 generic.go:356] "Generic (PLEG): container finished" podID="7176ec41-5592-4f76-b284-203843247cf8" containerID="e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b" exitCode=0
Oct 30 15:40:28.857161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.857128    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"
Oct 30 15:40:28.857161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.857138    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerDied","Data":"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"}
Oct 30 15:40:28.857443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.857181    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn" event={"ID":"7176ec41-5592-4f76-b284-203843247cf8","Type":"ContainerDied","Data":"98963b8a54ae986488e3a7331943e6521d5c0eff81dd83c47374c0d99ac559e2"}
Oct 30 15:40:28.857443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.857201    5064 scope.go:122] "RemoveContainer" containerID="e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"
Oct 30 15:40:28.867864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.867821    5064 scope.go:122] "RemoveContainer" containerID="e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"
Oct 30 15:40:28.878052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.878030    5064 scope.go:122] "RemoveContainer" containerID="e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"
Oct 30 15:40:28.878359 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:28.878340    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b\": container with ID starting with e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b not found: ID does not exist" containerID="e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"
Oct 30 15:40:28.878406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.878371    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b"} err="failed to get container status \"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b\": rpc error: code = NotFound desc = could not find container \"e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b\": container with ID starting with e71b8f8006e8f99fd73df64b4c09a077fee822ddf4987c8aa8bb975c7d37538b not found: ID does not exist"
Oct 30 15:40:28.878406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.878391    5064 scope.go:122] "RemoveContainer" containerID="e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"
Oct 30 15:40:28.878636 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:28.878612    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555\": container with ID starting with e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555 not found: ID does not exist" containerID="e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"
Oct 30 15:40:28.878680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.878643    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555"} err="failed to get container status \"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555\": rpc error: code = NotFound desc = could not find container \"e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555\": container with ID starting with e288ca58e41510f2b3191f103a2b569717ee4f59c03016f6d9e2a54448db0555 not found: ID does not exist"
Oct 30 15:40:28.916289 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.916248    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:40:28.929440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:28.929399    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-261ad0-openshift-default-86cccdfdb9-nmqdn"]
Oct 30 15:40:29.631326 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:29.631288    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:40:30.724599 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:30.724561    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7176ec41-5592-4f76-b284-203843247cf8" path="/var/lib/kubelet/pods/7176ec41-5592-4f76-b284-203843247cf8/volumes"
Oct 30 15:40:32.789125 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.789095    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:40:32.888614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.888570    5064 generic.go:356] "Generic (PLEG): container finished" podID="17c684b0-b899-4e29-862d-d190a492ffb7" containerID="479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406" exitCode=0
Oct 30 15:40:32.888792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.888682    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" event={"ID":"17c684b0-b899-4e29-862d-d190a492ffb7","Type":"ContainerDied","Data":"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"}
Oct 30 15:40:32.888792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.888738    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27" event={"ID":"17c684b0-b899-4e29-862d-d190a492ffb7","Type":"ContainerDied","Data":"0822f9390d023d8d00bda50c7e92f3fe137deee48cee74016998b111693a6146"}
Oct 30 15:40:32.889101 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.889073    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"
Oct 30 15:40:32.889231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.889209    5064 scope.go:122] "RemoveContainer" containerID="479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"
Oct 30 15:40:32.902727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.902695    5064 scope.go:122] "RemoveContainer" containerID="479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"
Oct 30 15:40:32.903118 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:32.903096    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406\": container with ID starting with 479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406 not found: ID does not exist" containerID="479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"
Oct 30 15:40:32.903208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.903130    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406"} err="failed to get container status \"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406\": rpc error: code = NotFound desc = could not find container \"479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406\": container with ID starting with 479372d7ddb4ab436281cc162715d54e61b5d9719ac5256ab5ed0ba77e00c406 not found: ID does not exist"
Oct 30 15:40:32.910197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910169    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910217    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910311    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910469    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910507    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:32.910611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910554    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:32.910611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910562    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:32.910611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910604    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910633    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910721 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910663    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910826 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910721    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910845    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\") pod \"17c684b0-b899-4e29-862d-d190a492ffb7\" (UID: \"17c684b0-b899-4e29-862d-d190a492ffb7\") "
Oct 30 15:40:32.910912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910874    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:32.911025 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.910939    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:32.911463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.911436    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c684b0-b899-4e29-862d-d190a492ffb7-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:32.911573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.911466    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:32.911573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.911481    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:32.911573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.911495    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:32.911573 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.911509    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:32.912905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.912876    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:32.912998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.912955    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "kube-api-access-zrswp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:32.912998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.912962    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:40:32.913105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:32.913005    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy" pod "17c684b0-b899-4e29-862d-d190a492ffb7" (UID: "17c684b0-b899-4e29-862d-d190a492ffb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:33.013160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.013064    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:33.013160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.013097    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c684b0-b899-4e29-862d-d190a492ffb7-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:33.013160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.013106    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c684b0-b899-4e29-862d-d190a492ffb7-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:33.013160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.013115    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrswp\" (UniqueName: \"kubernetes.io/projected/17c684b0-b899-4e29-862d-d190a492ffb7-kube-api-access-zrswp\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:33.037524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.037487    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:40:33.248286 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.248243    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:40:33.253904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.253871    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-cjn27"]
Oct 30 15:40:33.875259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.875222    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:40:33.912315 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:33.912281    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:40:34.724028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:34.723991    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" path="/var/lib/kubelet/pods/17c684b0-b899-4e29-862d-d190a492ffb7/volumes"
Oct 30 15:40:38.876406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:38.876358    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing" containerID="cri-o://4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f" gracePeriod=25
Oct 30 15:40:38.921908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:38.921039    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing" containerID="cri-o://ef7741ae60d5e350f001ebcffc4cbd538fbb583753bb421821b8f17648c53443" gracePeriod=25
Oct 30 15:40:41.623772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.623670    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:40:41.624891 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624866    5064 cpu_manager.go:414] "containerMap: removing container" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624892    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624910    5064 cpu_manager.go:414] "containerMap: removing container" podUID="27da83b4-c7c3-4336-8a33-03504738c856" containerName="maas-api"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624920    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27da83b4-c7c3-4336-8a33-03504738c856" containerName="maas-api"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624943    5064 cpu_manager.go:414] "containerMap: removing container" podUID="89094cc8-092a-4281-ace4-52ef6af5718c" containerName="maas-api"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624951    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89094cc8-092a-4281-ace4-52ef6af5718c" containerName="maas-api"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624967    5064 cpu_manager.go:414] "containerMap: removing container" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.624982    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625006    5064 cpu_manager.go:414] "containerMap: removing container" podUID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" containerName="maas-api"
Oct 30 15:40:41.625016 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625014    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" containerName="maas-api"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625026    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625035    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625048    5064 cpu_manager.go:414] "containerMap: removing container" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625055    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625066    5064 cpu_manager.go:414] "containerMap: removing container" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625075    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625084    5064 cpu_manager.go:414] "containerMap: removing container" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625091    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625119    5064 cpu_manager.go:414] "containerMap: removing container" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625127    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625150    5064 cpu_manager.go:414] "containerMap: removing container" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625157    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625168    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625175    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625189    5064 cpu_manager.go:414] "containerMap: removing container" podUID="6afd3f97-5981-4624-91b5-519fa4b031a8" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625196    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afd3f97-5981-4624-91b5-519fa4b031a8" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625212    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625219    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625232    5064 cpu_manager.go:414] "containerMap: removing container" podUID="728650fb-8a8b-439d-94d1-e8da3614699e" containerName="maas-api"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625239    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728650fb-8a8b-439d-94d1-e8da3614699e" containerName="maas-api"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625254    5064 cpu_manager.go:414] "containerMap: removing container" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerName="payload-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625260    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerName="payload-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625281    5064 cpu_manager.go:414] "containerMap: removing container" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625289    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625307    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625317    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625331    5064 cpu_manager.go:414] "containerMap: removing container" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerName="maas-api"
Oct 30 15:40:41.625480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625341    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625556    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625572    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="193959a9-6f5a-4d29-a1c0-be5433a7861f" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625583    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d274a787-7d52-46c4-ab60-e1db2d53b04f" containerName="payload-processing"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625593    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afd3f97-5981-4624-91b5-519fa4b031a8" containerName="payload-pre-processing"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625602    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625610    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="728650fb-8a8b-439d-94d1-e8da3614699e" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625620    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fc936e-4a4c-4628-b1f7-4649dd55dadf" containerName="payload-processing"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625630    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625638    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625650    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625660    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b507ad-1592-40a9-88e8-cf4ecb3a6698" containerName="payload-pre-processing"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625668    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd778804-48d0-4ef0-9b62-683b1ba8f171" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625680    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625688    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="27da83b4-c7c3-4336-8a33-03504738c856" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625695    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="89094cc8-092a-4281-ace4-52ef6af5718c" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625704    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c684b0-b899-4e29-862d-d190a492ffb7" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625715    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a78a95e-8f90-4712-ab7a-f93ee42fd54f" containerName="maas-api"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625724    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.626982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.625735    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb4f80a-42f1-4626-b736-cadd7c37453f" containerName="payload-pre-processing"
Oct 30 15:40:41.632378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.632343    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.648580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.648544    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:40:41.649028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.648999    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01c89d\""
Oct 30 15:40:41.664845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.664787    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmtq\" (UniqueName: \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.665036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.664872    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.766369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.766323    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmtq\" (UniqueName: \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.766579 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.766401    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.767276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.767235    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.770350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.770324    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:40:41.770957 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.770941    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.771022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.770957    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.771022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.770971    5064 cpu_manager.go:414] "containerMap: removing container" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.771022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.770977    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021bdc0-8e0a-4881-93c6-c9d70bdb2c6f" containerName="istio-proxy"
Oct 30 15:40:41.771022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.771003    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.771022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.771008    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7176ec41-5592-4f76-b284-203843247cf8" containerName="istio-proxy"
Oct 30 15:40:41.771244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.771122    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="69852b89-3caa-4c13-a40a-8bb194876a0e" containerName="istio-proxy"
Oct 30 15:40:41.771244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.771135    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1dd2eeb-dd55-47b1-908b-1ded7a67517a" containerName="istio-proxy"
Oct 30 15:40:41.771244 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.771145    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec42a1b4-40bd-4f6f-8ac7-13be20aab0f5" containerName="istio-proxy"
Oct 30 15:40:41.774634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.774615    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.794616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.794586    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmtq\" (UniqueName: \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.799321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.799279    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01c89d-dockercfg-pxcnd\""
Oct 30 15:40:41.802316 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.802288    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:40:41.867474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.867442    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz98\" (UniqueName: \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.867666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.867630    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.951010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.950905    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:41.969386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.969345    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.969644 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.969611    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz98\" (UniqueName: \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.970288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.970263    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:41.997483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:41.997449    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz98\" (UniqueName: \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\") pod \"payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:42.089754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.089714    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:42.109623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.108376    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:40:42.109623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.108747    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerName="maas-api" containerID="cri-o://20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8" gracePeriod=30
Oct 30 15:40:42.267066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.267008    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:40:42.277219 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.277177    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.165:8443/health\": dial tcp 10.132.0.165:8443: connect: connection refused"
Oct 30 15:40:42.278606 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:40:42.278517    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13c2f8d_93e3_4bcc_b50a_f58954c035cd.slice/crio-77741fabf5f4796ed23318c6eca80630d7baa57868a54ea48de25bb48f27626f WatchSource:0}: Error finding container 77741fabf5f4796ed23318c6eca80630d7baa57868a54ea48de25bb48f27626f: Status 404 returned error can't find the container with id 77741fabf5f4796ed23318c6eca80630d7baa57868a54ea48de25bb48f27626f
Oct 30 15:40:42.446410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.446379    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:40:42.449711 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:40:42.449680    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603b328d_1d3e_42a1_a4cb_1fdab718ae2c.slice/crio-3ef69568b37e9aa8a71949b4e448a4a1157ad4017ad5a88a2b89199d42313b8b WatchSource:0}: Error finding container 3ef69568b37e9aa8a71949b4e448a4a1157ad4017ad5a88a2b89199d42313b8b: Status 404 returned error can't find the container with id 3ef69568b37e9aa8a71949b4e448a4a1157ad4017ad5a88a2b89199d42313b8b
Oct 30 15:40:42.455856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.455821    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:40:42.605310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.605263    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\") pod \"f9b560ba-e043-4424-9fc4-962f42f0d321\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") "
Oct 30 15:40:42.605489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.605311    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\") pod \"f9b560ba-e043-4424-9fc4-962f42f0d321\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") "
Oct 30 15:40:42.605489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.605345    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\") pod \"f9b560ba-e043-4424-9fc4-962f42f0d321\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") "
Oct 30 15:40:42.605612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.605569    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\" (UniqueName: \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\") pod \"f9b560ba-e043-4424-9fc4-962f42f0d321\" (UID: \"f9b560ba-e043-4424-9fc4-962f42f0d321\") "
Oct 30 15:40:42.607079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.606847    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca" pod "f9b560ba-e043-4424-9fc4-962f42f0d321" (UID: "f9b560ba-e043-4424-9fc4-962f42f0d321"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:42.619867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.609219    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b560ba-e043-4424-9fc4-962f42f0d321-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:42.619867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.616349    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 15:40:42.619867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.616619    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" podUID="aabf3fae-02b3-494e-86e0-edf307a702a8" containerName="maas-api" containerID="cri-o://a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411" gracePeriod=30
Oct 30 15:40:42.625265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.624704    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls" pod "f9b560ba-e043-4424-9fc4-962f42f0d321" (UID: "f9b560ba-e043-4424-9fc4-962f42f0d321"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:42.625265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.624935    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs" pod "f9b560ba-e043-4424-9fc4-962f42f0d321" (UID: "f9b560ba-e043-4424-9fc4-962f42f0d321"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:42.625265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.625224    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8" pod "f9b560ba-e043-4424-9fc4-962f42f0d321" (UID: "f9b560ba-e043-4424-9fc4-962f42f0d321"). InnerVolumeSpecName "kube-api-access-n7mw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:42.710581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.710544    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mw8\" (UniqueName: \"kubernetes.io/projected/f9b560ba-e043-4424-9fc4-962f42f0d321-kube-api-access-n7mw8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:42.710581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.710580    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:42.710581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.710591    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b560ba-e043-4424-9fc4-962f42f0d321-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:42.930866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:42.930796    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 15:40:43.027490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.026422    5064 generic.go:356] "Generic (PLEG): container finished" podID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerID="20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8" exitCode=0
Oct 30 15:40:43.027490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.026521    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" event={"ID":"f9b560ba-e043-4424-9fc4-962f42f0d321","Type":"ContainerDied","Data":"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"}
Oct 30 15:40:43.027490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.026571    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx" event={"ID":"f9b560ba-e043-4424-9fc4-962f42f0d321","Type":"ContainerDied","Data":"b2d5e871baaf9af529c4d54053a4e96c79dd6fb9afcccdd0a0bcb5f995ead7da"}
Oct 30 15:40:43.027490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.026601    5064 scope.go:122] "RemoveContainer" containerID="20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"
Oct 30 15:40:43.027490 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.026750    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"
Oct 30 15:40:43.046961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.046877    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\" (UniqueName: \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\") pod \"aabf3fae-02b3-494e-86e0-edf307a702a8\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") "
Oct 30 15:40:43.048549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.048519    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") pod \"aabf3fae-02b3-494e-86e0-edf307a702a8\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") "
Oct 30 15:40:43.049872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.049485    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\") pod \"aabf3fae-02b3-494e-86e0-edf307a702a8\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") "
Oct 30 15:40:43.050559 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.050540    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\") pod \"aabf3fae-02b3-494e-86e0-edf307a702a8\" (UID: \"aabf3fae-02b3-494e-86e0-edf307a702a8\") "
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.061111    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" event={"ID":"603b328d-1d3e-42a1-a4cb-1fdab718ae2c","Type":"ContainerStarted","Data":"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.061175    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" event={"ID":"603b328d-1d3e-42a1-a4cb-1fdab718ae2c","Type":"ContainerStarted","Data":"3ef69568b37e9aa8a71949b4e448a4a1157ad4017ad5a88a2b89199d42313b8b"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.062036    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.064116    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" event={"ID":"d13c2f8d-93e3-4bcc-b50a-f58954c035cd","Type":"ContainerStarted","Data":"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.064162    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" event={"ID":"d13c2f8d-93e3-4bcc-b50a-f58954c035cd","Type":"ContainerStarted","Data":"77741fabf5f4796ed23318c6eca80630d7baa57868a54ea48de25bb48f27626f"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.064905    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.066587    5064 generic.go:356] "Generic (PLEG): container finished" podID="aabf3fae-02b3-494e-86e0-edf307a702a8" containerID="a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411" exitCode=0
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.066703    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" event={"ID":"aabf3fae-02b3-494e-86e0-edf307a702a8","Type":"ContainerDied","Data":"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.066746    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv" event={"ID":"aabf3fae-02b3-494e-86e0-edf307a702a8","Type":"ContainerDied","Data":"d230fa00bd5a14be7cfe1593444c715691124b6c1c2c1ca2065fd1c4ea7d1a7b"}
Oct 30 15:40:43.067061 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.066819    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"
Oct 30 15:40:43.081955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.081902    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca" pod "aabf3fae-02b3-494e-86e0-edf307a702a8" (UID: "aabf3fae-02b3-494e-86e0-edf307a702a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:43.115817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.115750    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:40:43.119001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.118969    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-bcd8d6-576857cdfd-flddx"]
Oct 30 15:40:43.126145 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.126107    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls" pod "aabf3fae-02b3-494e-86e0-edf307a702a8" (UID: "aabf3fae-02b3-494e-86e0-edf307a702a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:43.128437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.128401    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x" pod "aabf3fae-02b3-494e-86e0-edf307a702a8" (UID: "aabf3fae-02b3-494e-86e0-edf307a702a8"). InnerVolumeSpecName "kube-api-access-lr28x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:43.128582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.128411    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs" pod "aabf3fae-02b3-494e-86e0-edf307a702a8" (UID: "aabf3fae-02b3-494e-86e0-edf307a702a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:43.133683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.133661    5064 scope.go:122] "RemoveContainer" containerID="20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"
Oct 30 15:40:43.139243 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:43.139199    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8\": container with ID starting with 20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8 not found: ID does not exist" containerID="20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"
Oct 30 15:40:43.139375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.139245    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8"} err="failed to get container status \"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8\": rpc error: code = NotFound desc = could not find container \"20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8\": container with ID starting with 20c077f1539aedf29f9dcdac6365b117b1834c1261c34b4f6970cc6d1551efd8 not found: ID does not exist"
Oct 30 15:40:43.139375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.139273    5064 scope.go:122] "RemoveContainer" containerID="a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"
Oct 30 15:40:43.153037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.152786    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:43.153037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.152816    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr28x\" (UniqueName: \"kubernetes.io/projected/aabf3fae-02b3-494e-86e0-edf307a702a8-kube-api-access-lr28x\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:43.153037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.152843    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabf3fae-02b3-494e-86e0-edf307a702a8-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:43.153037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.152859    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabf3fae-02b3-494e-86e0-edf307a702a8-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:43.156644 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.156590    5064 scope.go:122] "RemoveContainer" containerID="a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"
Oct 30 15:40:43.157109 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:43.157070    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411\": container with ID starting with a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411 not found: ID does not exist" containerID="a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"
Oct 30 15:40:43.157232 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.157111    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411"} err="failed to get container status \"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411\": rpc error: code = NotFound desc = could not find container \"a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411\": container with ID starting with a6770f1f55e57946002888db073fa3d3e408d5b0970d9c14979868a04007e411 not found: ID does not exist"
Oct 30 15:40:43.179532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.179451    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" podStartSLOduration=2.179434221 podStartE2EDuration="2.179434221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:43.175784815 +0000 UTC m=+3192.617675529" watchObservedRunningTime="2026-10-30 15:40:43.179434221 +0000 UTC m=+3192.621324943"
Oct 30 15:40:43.249736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.249673    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" podStartSLOduration=2.249655382 podStartE2EDuration="2.249655382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:43.247876587 +0000 UTC m=+3192.689767325" watchObservedRunningTime="2026-10-30 15:40:43.249655382 +0000 UTC m=+3192.691546100"
Oct 30 15:40:43.438914 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.438874    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 15:40:43.444759 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:43.444723    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2c8d89-f4bf95dcd-ksgkv"]
Oct 30 15:40:44.731349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:44.731306    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aabf3fae-02b3-494e-86e0-edf307a702a8" path="/var/lib/kubelet/pods/aabf3fae-02b3-494e-86e0-edf307a702a8/volumes"
Oct 30 15:40:44.732107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:44.732080    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" path="/var/lib/kubelet/pods/f9b560ba-e043-4424-9fc4-962f42f0d321/volumes"
Oct 30 15:40:46.734730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:46.734685    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:40:46.734730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:46.734711    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:40:48.059586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.059540    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" containerID="cri-o://a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49" gracePeriod=45
Oct 30 15:40:48.341371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.341340    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:40:48.421355 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421317    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.421560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421375    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.421560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421502    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.421671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421652    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\" (UniqueName: \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.421740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421689    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:48.421740 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421712    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\" (UniqueName: \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.421897 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.421789    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\") pod \"dad20a01-4a33-461c-9919-fbcbda8f6158\" (UID: \"dad20a01-4a33-461c-9919-fbcbda8f6158\") "
Oct 30 15:40:48.422049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.422023    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:48.422472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.422430    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:48.422472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.422454    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:48.422651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.422515    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:48.425768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.425734    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:48.425900 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.425764    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "kube-api-access-r9sqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:48.425947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.425920    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm" pod "dad20a01-4a33-461c-9919-fbcbda8f6158" (UID: "dad20a01-4a33-461c-9919-fbcbda8f6158"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:48.523899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.523863    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:48.523899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.523895    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9sqj\" (UniqueName: \"kubernetes.io/projected/dad20a01-4a33-461c-9919-fbcbda8f6158-kube-api-access-r9sqj\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:48.523899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.523906    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad20a01-4a33-461c-9919-fbcbda8f6158-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:48.524134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:48.523916    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad20a01-4a33-461c-9919-fbcbda8f6158-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:49.119713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.119680    5064 generic.go:356] "Generic (PLEG): container finished" podID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerID="a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49" exitCode=0
Oct 30 15:40:49.120176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.119729    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"
Oct 30 15:40:49.120176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.119767    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" event={"ID":"dad20a01-4a33-461c-9919-fbcbda8f6158","Type":"ContainerDied","Data":"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"}
Oct 30 15:40:49.120176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.119817    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" event={"ID":"dad20a01-4a33-461c-9919-fbcbda8f6158","Type":"ContainerDied","Data":"5f15788b3d0a2e1e8661252d32aac54503fb0f10211425442236560c9c4e133c"}
Oct 30 15:40:49.120176 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.119867    5064 scope.go:122] "RemoveContainer" containerID="a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"
Oct 30 15:40:49.129888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.129856    5064 scope.go:122] "RemoveContainer" containerID="a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"
Oct 30 15:40:49.130210 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:49.130188    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49\": container with ID starting with a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49 not found: ID does not exist" containerID="a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"
Oct 30 15:40:49.130297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.130219    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49"} err="failed to get container status \"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49\": rpc error: code = NotFound desc = could not find container \"a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49\": container with ID starting with a83f113ba18c328d9d13827983401e0b393edc048af2c0c1907f6115c157ce49 not found: ID does not exist"
Oct 30 15:40:49.183271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.183232    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:40:49.196913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.196879    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd"]
Oct 30 15:40:49.307875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:49.307785    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-2-w3-357948-kserve-679d8584d9-kbxbd" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:40:50.726277 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:50.726239    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" path="/var/lib/kubelet/pods/dad20a01-4a33-461c-9919-fbcbda8f6158/volumes"
Oct 30 15:40:51.413576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:51.413536    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:40:51.414131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:51.414088    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy" containerID="cri-o://98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74" gracePeriod=30
Oct 30 15:40:52.465745 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.465701    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:40:52.466302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.465979    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" podUID="de1735e4-781b-437e-8b5b-3b1ecb784e13" containerName="maas-api" containerID="cri-o://dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5" gracePeriod=30
Oct 30 15:40:52.804993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.804952    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:40:52.878882 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.878842    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:40:52.879370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.879333    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" podUID="55005f68-0a05-4bc9-aabf-fcd422046bd6" containerName="maas-api" containerID="cri-o://fafd1a50ea5232f65f16739a0921343ea3c923981edc864cf437c44d2b55d517" gracePeriod=30
Oct 30 15:40:52.894615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.894578    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\") pod \"de1735e4-781b-437e-8b5b-3b1ecb784e13\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") "
Oct 30 15:40:52.894940 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.894924    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\") pod \"de1735e4-781b-437e-8b5b-3b1ecb784e13\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") "
Oct 30 15:40:52.895140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.895124    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\") pod \"de1735e4-781b-437e-8b5b-3b1ecb784e13\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") "
Oct 30 15:40:52.895329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.895313    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\" (UniqueName: \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\") pod \"de1735e4-781b-437e-8b5b-3b1ecb784e13\" (UID: \"de1735e4-781b-437e-8b5b-3b1ecb784e13\") "
Oct 30 15:40:52.906294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.905578    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca" pod "de1735e4-781b-437e-8b5b-3b1ecb784e13" (UID: "de1735e4-781b-437e-8b5b-3b1ecb784e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:52.933669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.924526    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs" pod "de1735e4-781b-437e-8b5b-3b1ecb784e13" (UID: "de1735e4-781b-437e-8b5b-3b1ecb784e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:52.942198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.942151    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls" pod "de1735e4-781b-437e-8b5b-3b1ecb784e13" (UID: "de1735e4-781b-437e-8b5b-3b1ecb784e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:52.969149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.969070    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt" pod "de1735e4-781b-437e-8b5b-3b1ecb784e13" (UID: "de1735e4-781b-437e-8b5b-3b1ecb784e13"). InnerVolumeSpecName "kube-api-access-clrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:52.997197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.997153    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:52.997197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.997197    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1735e4-781b-437e-8b5b-3b1ecb784e13-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:52.997521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.997217    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1735e4-781b-437e-8b5b-3b1ecb784e13-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:52.997521 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:52.997233    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrgt\" (UniqueName: \"kubernetes.io/projected/de1735e4-781b-437e-8b5b-3b1ecb784e13-kube-api-access-clrgt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:53.152932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.152779    5064 generic.go:356] "Generic (PLEG): container finished" podID="55005f68-0a05-4bc9-aabf-fcd422046bd6" containerID="fafd1a50ea5232f65f16739a0921343ea3c923981edc864cf437c44d2b55d517" exitCode=0
Oct 30 15:40:53.152932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.152816    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" event={"ID":"55005f68-0a05-4bc9-aabf-fcd422046bd6","Type":"ContainerDied","Data":"fafd1a50ea5232f65f16739a0921343ea3c923981edc864cf437c44d2b55d517"}
Oct 30 15:40:53.155076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.155042    5064 generic.go:356] "Generic (PLEG): container finished" podID="de1735e4-781b-437e-8b5b-3b1ecb784e13" containerID="dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5" exitCode=0
Oct 30 15:40:53.155250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.155120    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" event={"ID":"de1735e4-781b-437e-8b5b-3b1ecb784e13","Type":"ContainerDied","Data":"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"}
Oct 30 15:40:53.155250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.155164    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv" event={"ID":"de1735e4-781b-437e-8b5b-3b1ecb784e13","Type":"ContainerDied","Data":"d59a3c38e763ed10ed603b6cdde902b8721bd0f4cab09e242b3001cb325d4200"}
Oct 30 15:40:53.155250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.155194    5064 scope.go:122] "RemoveContainer" containerID="dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"
Oct 30 15:40:53.155457 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.155343    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"
Oct 30 15:40:53.175085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.174955    5064 scope.go:122] "RemoveContainer" containerID="dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"
Oct 30 15:40:53.175938 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:53.175906    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5\": container with ID starting with dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5 not found: ID does not exist" containerID="dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"
Oct 30 15:40:53.176090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.175952    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5"} err="failed to get container status \"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5\": rpc error: code = NotFound desc = could not find container \"dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5\": container with ID starting with dd1bef69ea35d5f2e36ee19d5072320ced56f8f5c004e079be032b41e67d9ab5 not found: ID does not exist"
Oct 30 15:40:53.193015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.192973    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:40:53.219965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.219927    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:40:53.228863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.228813    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3f20f9-7c94466bc-j9hsv"]
Oct 30 15:40:53.301206 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301171    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\") pod \"55005f68-0a05-4bc9-aabf-fcd422046bd6\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") "
Oct 30 15:40:53.301394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301223    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\" (UniqueName: \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\") pod \"55005f68-0a05-4bc9-aabf-fcd422046bd6\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") "
Oct 30 15:40:53.301394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301268    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\") pod \"55005f68-0a05-4bc9-aabf-fcd422046bd6\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") "
Oct 30 15:40:53.301394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301352    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\") pod \"55005f68-0a05-4bc9-aabf-fcd422046bd6\" (UID: \"55005f68-0a05-4bc9-aabf-fcd422046bd6\") "
Oct 30 15:40:53.301591 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301557    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca" pod "55005f68-0a05-4bc9-aabf-fcd422046bd6" (UID: "55005f68-0a05-4bc9-aabf-fcd422046bd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:53.302015 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.301964    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55005f68-0a05-4bc9-aabf-fcd422046bd6-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:53.303484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.303459    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls" pod "55005f68-0a05-4bc9-aabf-fcd422046bd6" (UID: "55005f68-0a05-4bc9-aabf-fcd422046bd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:53.303949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.303921    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs" pod "55005f68-0a05-4bc9-aabf-fcd422046bd6" (UID: "55005f68-0a05-4bc9-aabf-fcd422046bd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:40:53.304067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.304023    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt" pod "55005f68-0a05-4bc9-aabf-fcd422046bd6" (UID: "55005f68-0a05-4bc9-aabf-fcd422046bd6"). InnerVolumeSpecName "kube-api-access-w9nvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:53.403011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.402976    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:53.403011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.403008    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nvt\" (UniqueName: \"kubernetes.io/projected/55005f68-0a05-4bc9-aabf-fcd422046bd6-kube-api-access-w9nvt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:53.403242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:53.403023    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55005f68-0a05-4bc9-aabf-fcd422046bd6-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:54.164356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.164326    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"
Oct 30 15:40:54.164797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.164360    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f" event={"ID":"55005f68-0a05-4bc9-aabf-fcd422046bd6","Type":"ContainerDied","Data":"c182c9e164b5797e7cd891145904045dad7e05e033c8bfd07c4118213a6e58a8"}
Oct 30 15:40:54.164797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.164426    5064 scope.go:122] "RemoveContainer" containerID="fafd1a50ea5232f65f16739a0921343ea3c923981edc864cf437c44d2b55d517"
Oct 30 15:40:54.221008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.220941    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:40:54.233660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.233629    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0644a3-848cfb95bc-9q82f"]
Oct 30 15:40:54.723095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.723059    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55005f68-0a05-4bc9-aabf-fcd422046bd6" path="/var/lib/kubelet/pods/55005f68-0a05-4bc9-aabf-fcd422046bd6/volumes"
Oct 30 15:40:54.723627 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:54.723608    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1735e4-781b-437e-8b5b-3b1ecb784e13" path="/var/lib/kubelet/pods/de1735e4-781b-437e-8b5b-3b1ecb784e13/volumes"
Oct 30 15:40:55.082943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:55.082852    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:40:55.082943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:55.082927    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:40:56.699781 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.699709    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:40:56.734998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.734961    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:40:56.735188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.735071    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:40:56.772653 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772617    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.772876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772702    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.772876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772746    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.772876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772805    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.772876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772871    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.773141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772935    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.773141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.772968    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.773141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.773060    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.773141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.773126    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\") pod \"fe822e50-9c5e-4233-893a-20179742182d\" (UID: \"fe822e50-9c5e-4233-893a-20179742182d\") "
Oct 30 15:40:56.773511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.773485    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:56.773808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.773779    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:56.774034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.773794    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe822e50-9c5e-4233-893a-20179742182d-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.774461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.774439    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:56.774623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.774536    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:56.774751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.774699    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:56.777099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.777065    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "kube-api-access-h9xgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:56.777393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.777303    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:56.777393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.777350    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:40:56.777557 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.777503    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy" pod "fe822e50-9c5e-4233-893a-20179742182d" (UID: "fe822e50-9c5e-4233-893a-20179742182d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:40:56.875670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875635    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875667    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875678    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xgx\" (UniqueName: \"kubernetes.io/projected/fe822e50-9c5e-4233-893a-20179742182d-kube-api-access-h9xgx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875689    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe822e50-9c5e-4233-893a-20179742182d-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875698    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875711    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875721    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:56.875946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:56.875729    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe822e50-9c5e-4233-893a-20179742182d-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:57.188003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.187964    5064 generic.go:356] "Generic (PLEG): container finished" podID="fe822e50-9c5e-4233-893a-20179742182d" containerID="98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74" exitCode=0
Oct 30 15:40:57.188197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.188022    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"
Oct 30 15:40:57.188197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.188048    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" event={"ID":"fe822e50-9c5e-4233-893a-20179742182d","Type":"ContainerDied","Data":"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"}
Oct 30 15:40:57.188197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.188108    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2" event={"ID":"fe822e50-9c5e-4233-893a-20179742182d","Type":"ContainerDied","Data":"df2c3856369c3748130c186147da462f0625fffe97722f68cfaaf6771c40fe1b"}
Oct 30 15:40:57.188197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.188131    5064 scope.go:122] "RemoveContainer" containerID="98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"
Oct 30 15:40:57.199747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.199728    5064 scope.go:122] "RemoveContainer" containerID="98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"
Oct 30 15:40:57.200192 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:57.200168    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74\": container with ID starting with 98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74 not found: ID does not exist" containerID="98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"
Oct 30 15:40:57.200300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.200199    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74"} err="failed to get container status \"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74\": rpc error: code = NotFound desc = could not find container \"98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74\": container with ID starting with 98fed363c91ee9c11653c3a353d9c29a3268ba8a22136758654746fc71a68f74 not found: ID does not exist"
Oct 30 15:40:57.245744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.245705    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:40:57.258765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:57.258733    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-rhxx2"]
Oct 30 15:40:58.724995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:58.724955    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe822e50-9c5e-4233-893a-20179742182d" path="/var/lib/kubelet/pods/fe822e50-9c5e-4233-893a-20179742182d/volumes"
Oct 30 15:40:59.198083 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.198057    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:40:59.204877 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.204848    5064 generic.go:356] "Generic (PLEG): container finished" podID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerID="4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f" exitCode=0
Oct 30 15:40:59.205057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.204903    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"
Oct 30 15:40:59.205057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.204943    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" event={"ID":"2b0027ba-5a92-4c25-940a-ce550abcfa9f","Type":"ContainerDied","Data":"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"}
Oct 30 15:40:59.205057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.205014    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf" event={"ID":"2b0027ba-5a92-4c25-940a-ce550abcfa9f","Type":"ContainerDied","Data":"3bfffbf804ae940c24589f0c159f2aca8c7d9d690a5ff2993a1cbdb66fe26575"}
Oct 30 15:40:59.205057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.205044    5064 scope.go:122] "RemoveContainer" containerID="4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"
Oct 30 15:40:59.206629 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.206606    5064 generic.go:356] "Generic (PLEG): container finished" podID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerID="ef7741ae60d5e350f001ebcffc4cbd538fbb583753bb421821b8f17648c53443" exitCode=0
Oct 30 15:40:59.206732 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.206687    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" event={"ID":"50e99d3d-66dc-42bd-96db-8b5ae46c592f","Type":"ContainerDied","Data":"ef7741ae60d5e350f001ebcffc4cbd538fbb583753bb421821b8f17648c53443"}
Oct 30 15:40:59.216947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.216928    5064 scope.go:122] "RemoveContainer" containerID="4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"
Oct 30 15:40:59.217218 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:59.217196    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f\": container with ID starting with 4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f not found: ID does not exist" containerID="4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"
Oct 30 15:40:59.217288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.217224    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f"} err="failed to get container status \"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f\": rpc error: code = NotFound desc = could not find container \"4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f\": container with ID starting with 4505494d6366559cd3f42cbb9a446c95ae5a8b0e7154ca679b4f8cfcf9d20c8f not found: ID does not exist"
Oct 30 15:40:59.252895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.252867    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:40:59.310037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.309936    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\" (UniqueName: \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\") pod \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") "
Oct 30 15:40:59.310264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.310042    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\" (UniqueName: \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\") pod \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") "
Oct 30 15:40:59.310264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.310214    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\") pod \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\" (UID: \"50e99d3d-66dc-42bd-96db-8b5ae46c592f\") "
Oct 30 15:40:59.310264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.310250    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\") pod \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\" (UID: \"2b0027ba-5a92-4c25-940a-ce550abcfa9f\") "
Oct 30 15:40:59.311489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.311462    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume" pod "2b0027ba-5a92-4c25-940a-ce550abcfa9f" (UID: "2b0027ba-5a92-4c25-940a-ce550abcfa9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:59.311884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.311803    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume" pod "50e99d3d-66dc-42bd-96db-8b5ae46c592f" (UID: "50e99d3d-66dc-42bd-96db-8b5ae46c592f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:40:59.314428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.314384    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9" pod "2b0027ba-5a92-4c25-940a-ce550abcfa9f" (UID: "2b0027ba-5a92-4c25-940a-ce550abcfa9f"). InnerVolumeSpecName "kube-api-access-2ddw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:59.315002 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.314972    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs" pod "50e99d3d-66dc-42bd-96db-8b5ae46c592f" (UID: "50e99d3d-66dc-42bd-96db-8b5ae46c592f"). InnerVolumeSpecName "kube-api-access-m2hgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:40:59.412787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.412748    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hgs\" (UniqueName: \"kubernetes.io/projected/50e99d3d-66dc-42bd-96db-8b5ae46c592f-kube-api-access-m2hgs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:59.412787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.412782    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddw9\" (UniqueName: \"kubernetes.io/projected/2b0027ba-5a92-4c25-940a-ce550abcfa9f-kube-api-access-2ddw9\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:59.412787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.412794    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e99d3d-66dc-42bd-96db-8b5ae46c592f-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:59.413076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.412803    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0027ba-5a92-4c25-940a-ce550abcfa9f-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:40:59.553466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.553424    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:40:59.557633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:40:59.557604    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bcd8d6-5fccdf9dbb-5w7sf"]
Oct 30 15:40:59.631083 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:40:59.631044    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:41:00.214812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.214724    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"
Oct 30 15:41:00.214812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.214756    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l" event={"ID":"50e99d3d-66dc-42bd-96db-8b5ae46c592f","Type":"ContainerDied","Data":"26a05eb4c990948900745553f6628660d3db18cba679e4e5308784af2cc9755c"}
Oct 30 15:41:00.215339 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.214842    5064 scope.go:122] "RemoveContainer" containerID="ef7741ae60d5e350f001ebcffc4cbd538fbb583753bb421821b8f17648c53443"
Oct 30 15:41:00.270715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.270675    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:41:00.284224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.284183    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bcd8d6-64b77c6679-j5k6l"]
Oct 30 15:41:00.729757 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.729715    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" path="/var/lib/kubelet/pods/2b0027ba-5a92-4c25-940a-ce550abcfa9f/volumes"
Oct 30 15:41:00.730455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.730430    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" path="/var/lib/kubelet/pods/50e99d3d-66dc-42bd-96db-8b5ae46c592f/volumes"
Oct 30 15:41:00.927585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.927549    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:41:00.927814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:00.927790    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy" containerID="cri-o://c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1" gracePeriod=30
Oct 30 15:41:01.639517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:01.639480    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:41:01.640053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:01.639693    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" containerID="cri-o://b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8" gracePeriod=30
Oct 30 15:41:05.165399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:05.165353    5064 patch_prober.go:29] interesting pod/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:41:05.165812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:05.165415    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:41:06.195026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.194952    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:41:06.260865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.260799    5064 generic.go:356] "Generic (PLEG): container finished" podID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerID="c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1" exitCode=0
Oct 30 15:41:06.261050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.260937    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerDied","Data":"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"}
Oct 30 15:41:06.261050 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.261036    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh" event={"ID":"d275e859-a3c6-4d17-97b9-3bd90f0aa285","Type":"ContainerDied","Data":"aa04161a28e64970e9807b5486884fa53468b69c34de449f5f8cb30644235858"}
Oct 30 15:41:06.261161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.261071    5064 scope.go:122] "RemoveContainer" containerID="c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"
Oct 30 15:41:06.261231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.261218    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"
Oct 30 15:41:06.274762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.274739    5064 scope.go:122] "RemoveContainer" containerID="3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"
Oct 30 15:41:06.285093 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.285069    5064 scope.go:122] "RemoveContainer" containerID="c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"
Oct 30 15:41:06.285431 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:06.285407    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1\": container with ID starting with c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1 not found: ID does not exist" containerID="c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"
Oct 30 15:41:06.285513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.285441    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1"} err="failed to get container status \"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1\": rpc error: code = NotFound desc = could not find container \"c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1\": container with ID starting with c75116ebef48e2f7f433ebf0a1a7cada46b95f703e4708fb316eea2ee9d42de1 not found: ID does not exist"
Oct 30 15:41:06.285513 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.285462    5064 scope.go:122] "RemoveContainer" containerID="3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"
Oct 30 15:41:06.285737 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:06.285710    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb\": container with ID starting with 3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb not found: ID does not exist" containerID="3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"
Oct 30 15:41:06.285805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.285747    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb"} err="failed to get container status \"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb\": rpc error: code = NotFound desc = could not find container \"3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb\": container with ID starting with 3d1e5d4897fc2a6b5358efc6b972b63ed5eb869bedda3d335ac884a21ccd2ebb not found: ID does not exist"
Oct 30 15:41:06.332162 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332128    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332172    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332288    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332364    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332395    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332452    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332501    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:06.332531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332516    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332542    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:06.332600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332581    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:06.332683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332624    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332658    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\") pod \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\" (UID: \"d275e859-a3c6-4d17-97b9-3bd90f0aa285\") "
Oct 30 15:41:06.332799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.332767    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:06.333048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333018    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:06.333288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333266    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.333369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333293    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.333369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333309    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.333369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333323    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.333369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.333335    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.334779 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.334756    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:06.334895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.334873    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "kube-api-access-jlh4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:06.335078 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.335058    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:06.335140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.335082    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token" pod "d275e859-a3c6-4d17-97b9-3bd90f0aa285" (UID: "d275e859-a3c6-4d17-97b9-3bd90f0aa285"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:06.434482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.434441    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.434482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.434474    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.434482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.434483    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d275e859-a3c6-4d17-97b9-3bd90f0aa285-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.434482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.434492    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlh4l\" (UniqueName: \"kubernetes.io/projected/d275e859-a3c6-4d17-97b9-3bd90f0aa285-kube-api-access-jlh4l\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:06.618540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.618501    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:41:06.621479 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.621448    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-nrsbh"]
Oct 30 15:41:06.724247 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.724210    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" path="/var/lib/kubelet/pods/d275e859-a3c6-4d17-97b9-3bd90f0aa285/volumes"
Oct 30 15:41:06.894407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:06.894381    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:41:07.041563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041524    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041642    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041669    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041696    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041724    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041825    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.041947 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041860    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:07.042049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.041916    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.042049 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042026    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.042412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042001    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:07.042412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042065    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:07.042412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042067    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\") pod \"227280a0-0c75-47d4-81c2-8c2807a4ea26\" (UID: \"227280a0-0c75-47d4-81c2-8c2807a4ea26\") "
Oct 30 15:41:07.042412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042151    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:07.042412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042251    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:07.042744 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042720    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.042791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042745    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.042791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042760    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.042791 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042775    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227280a0-0c75-47d4-81c2-8c2807a4ea26-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.042924 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.042790    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.044140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.044108    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:07.044397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.044377    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "kube-api-access-sg4zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:07.044614 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.044571    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:07.044707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.044636    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy" pod "227280a0-0c75-47d4-81c2-8c2807a4ea26" (UID: "227280a0-0c75-47d4-81c2-8c2807a4ea26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:07.144319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.144227    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.144319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.144262    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.144319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.144275    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227280a0-0c75-47d4-81c2-8c2807a4ea26-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.144319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.144284    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4zs\" (UniqueName: \"kubernetes.io/projected/227280a0-0c75-47d4-81c2-8c2807a4ea26-kube-api-access-sg4zs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:07.269718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.269682    5064 generic.go:356] "Generic (PLEG): container finished" podID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerID="b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8" exitCode=0
Oct 30 15:41:07.270194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.269725    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerDied","Data":"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"}
Oct 30 15:41:07.270194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.269733    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"
Oct 30 15:41:07.270194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.269767    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v" event={"ID":"227280a0-0c75-47d4-81c2-8c2807a4ea26","Type":"ContainerDied","Data":"bd6dee10b32d719856c7d9d4b1455a1a48ffb0a6fe207974b5737f8545c8a7d3"}
Oct 30 15:41:07.270194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.269788    5064 scope.go:122] "RemoveContainer" containerID="b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"
Oct 30 15:41:07.280756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.280734    5064 scope.go:122] "RemoveContainer" containerID="5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"
Oct 30 15:41:07.290319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.290297    5064 scope.go:122] "RemoveContainer" containerID="b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"
Oct 30 15:41:07.290617 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:07.290590    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8\": container with ID starting with b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8 not found: ID does not exist" containerID="b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"
Oct 30 15:41:07.290712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.290627    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8"} err="failed to get container status \"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8\": rpc error: code = NotFound desc = could not find container \"b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8\": container with ID starting with b86809b863002f1f3321c3448f27644266a774c703c78ef39e699f2b1f94bbe8 not found: ID does not exist"
Oct 30 15:41:07.290712 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.290651    5064 scope.go:122] "RemoveContainer" containerID="5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"
Oct 30 15:41:07.290945 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:07.290922    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54\": container with ID starting with 5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54 not found: ID does not exist" containerID="5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"
Oct 30 15:41:07.291080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.290952    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54"} err="failed to get container status \"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54\": rpc error: code = NotFound desc = could not find container \"5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54\": container with ID starting with 5171cd850a7f99081d7cf1a52e081442b5df9c15b272d21f80000785748dcc54 not found: ID does not exist"
Oct 30 15:41:07.328026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.327985    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:41:07.341841 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:07.341796    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0644a3-openshift-default-5d5dbc865f-26n2v"]
Oct 30 15:41:08.723095 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:08.723062    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" path="/var/lib/kubelet/pods/227280a0-0c75-47d4-81c2-8c2807a4ea26/volumes"
Oct 30 15:41:12.009231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:12.009193    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:41:12.009871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:12.009502    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy" containerID="cri-o://d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa" gracePeriod=30
Oct 30 15:41:13.686658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:13.686606    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:41:13.687149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:13.686875    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" podUID="9ba81452-32ec-4672-9892-2dbbabe73285" containerName="maas-api" containerID="cri-o://f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869" gracePeriod=30
Oct 30 15:41:13.947606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:13.947535    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:41:14.144855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.134227    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\" (UniqueName: \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\") pod \"9ba81452-32ec-4672-9892-2dbbabe73285\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") "
Oct 30 15:41:14.144855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.134287    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\") pod \"9ba81452-32ec-4672-9892-2dbbabe73285\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") "
Oct 30 15:41:14.144855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.134351    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\") pod \"9ba81452-32ec-4672-9892-2dbbabe73285\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") "
Oct 30 15:41:14.144855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.134481    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") pod \"9ba81452-32ec-4672-9892-2dbbabe73285\" (UID: \"9ba81452-32ec-4672-9892-2dbbabe73285\") "
Oct 30 15:41:14.159858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.157240    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca" pod "9ba81452-32ec-4672-9892-2dbbabe73285" (UID: "9ba81452-32ec-4672-9892-2dbbabe73285"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:14.161000 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.160940    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 15:41:14.163733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.161418    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" podUID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" containerName="maas-api" containerID="cri-o://c126022f4203b19adba7dff9fae20c387094666ba717085a0b7feff96906fbcb" gracePeriod=30
Oct 30 15:41:14.212951 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.212814    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls" pod "9ba81452-32ec-4672-9892-2dbbabe73285" (UID: "9ba81452-32ec-4672-9892-2dbbabe73285"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:14.213545 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.213509    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95" pod "9ba81452-32ec-4672-9892-2dbbabe73285" (UID: "9ba81452-32ec-4672-9892-2dbbabe73285"). InnerVolumeSpecName "kube-api-access-5tk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:14.214141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.214113    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs" pod "9ba81452-32ec-4672-9892-2dbbabe73285" (UID: "9ba81452-32ec-4672-9892-2dbbabe73285"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:14.236600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.236557    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.236600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.236603    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk95\" (UniqueName: \"kubernetes.io/projected/9ba81452-32ec-4672-9892-2dbbabe73285-kube-api-access-5tk95\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.236905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.236620    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba81452-32ec-4672-9892-2dbbabe73285-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.236905 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.236635    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba81452-32ec-4672-9892-2dbbabe73285-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.355497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.355464    5064 generic.go:356] "Generic (PLEG): container finished" podID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" containerID="c126022f4203b19adba7dff9fae20c387094666ba717085a0b7feff96906fbcb" exitCode=0
Oct 30 15:41:14.355679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.355537    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" event={"ID":"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9","Type":"ContainerDied","Data":"c126022f4203b19adba7dff9fae20c387094666ba717085a0b7feff96906fbcb"}
Oct 30 15:41:14.357289 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.357259    5064 generic.go:356] "Generic (PLEG): container finished" podID="9ba81452-32ec-4672-9892-2dbbabe73285" containerID="f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869" exitCode=0
Oct 30 15:41:14.357411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.357322    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"
Oct 30 15:41:14.357411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.357342    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" event={"ID":"9ba81452-32ec-4672-9892-2dbbabe73285","Type":"ContainerDied","Data":"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"}
Oct 30 15:41:14.357411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.357394    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2" event={"ID":"9ba81452-32ec-4672-9892-2dbbabe73285","Type":"ContainerDied","Data":"67ebe50941fdc027bb94b5900c7f5b8dbf1be1725a01a092fe9c0bfb16e6bd47"}
Oct 30 15:41:14.357565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.357417    5064 scope.go:122] "RemoveContainer" containerID="f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"
Oct 30 15:41:14.371804 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.371783    5064 scope.go:122] "RemoveContainer" containerID="f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"
Oct 30 15:41:14.372174 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:14.372129    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869\": container with ID starting with f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869 not found: ID does not exist" containerID="f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"
Oct 30 15:41:14.372256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.372180    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869"} err="failed to get container status \"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869\": rpc error: code = NotFound desc = could not find container \"f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869\": container with ID starting with f5112858644919cf610cbb2603f5b6100d595c08a2c40aa1460bfd4b08773869 not found: ID does not exist"
Oct 30 15:41:14.415363 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.415124    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:41:14.421075 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.421045    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5a5948-f465fd8c6-xpfp2"]
Oct 30 15:41:14.470325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.470298    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 15:41:14.641208 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.641173    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\") pod \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") "
Oct 30 15:41:14.641386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.641255    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\") pod \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") "
Oct 30 15:41:14.641386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.641301    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\" (UniqueName: \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\") pod \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") "
Oct 30 15:41:14.641480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.641401    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\") pod \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\" (UID: \"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9\") "
Oct 30 15:41:14.641704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.641668    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca" pod "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" (UID: "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:14.643717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.643690    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l" pod "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" (UID: "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9"). InnerVolumeSpecName "kube-api-access-2bc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:14.644032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.644007    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls" pod "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" (UID: "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:14.644126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.644031    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs" pod "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" (UID: "377dfec6-1315-4dd7-81c3-32fbfdc9e6b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:14.759140 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.759092    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba81452-32ec-4672-9892-2dbbabe73285" path="/var/lib/kubelet/pods/9ba81452-32ec-4672-9892-2dbbabe73285/volumes"
Oct 30 15:41:14.764254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.764202    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.764254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.764257    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.764565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.764273    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.764565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.764287    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bc8l\" (UniqueName: \"kubernetes.io/projected/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9-kube-api-access-2bc8l\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:14.876420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.876379    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:41:14.876623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.876596    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerName="payload-processing" containerID="cri-o://d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72" gracePeriod=30
Oct 30 15:41:14.897056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.897001    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:41:14.897491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:14.897462    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerName="payload-pre-processing" containerID="cri-o://54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406" gracePeriod=30
Oct 30 15:41:15.083088 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.081713    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.197:9004: connect: connection refused"
Oct 30 15:41:15.283676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.283651    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:41:15.287026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.287002    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:41:15.368112 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.368075    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"
Oct 30 15:41:15.368332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.368079    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k" event={"ID":"377dfec6-1315-4dd7-81c3-32fbfdc9e6b9","Type":"ContainerDied","Data":"56b6a314286b18c5f803054e4e7522957b07a56b64031bb1438bf0e89180486c"}
Oct 30 15:41:15.368332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.368274    5064 scope.go:122] "RemoveContainer" containerID="c126022f4203b19adba7dff9fae20c387094666ba717085a0b7feff96906fbcb"
Oct 30 15:41:15.371045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.370999    5064 generic.go:356] "Generic (PLEG): container finished" podID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerID="d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72" exitCode=0
Oct 30 15:41:15.371211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.371194    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"
Oct 30 15:41:15.371300 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.371221    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" event={"ID":"603b328d-1d3e-42a1-a4cb-1fdab718ae2c","Type":"ContainerDied","Data":"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"}
Oct 30 15:41:15.371370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.371297    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" event={"ID":"603b328d-1d3e-42a1-a4cb-1fdab718ae2c","Type":"ContainerDied","Data":"3ef69568b37e9aa8a71949b4e448a4a1157ad4017ad5a88a2b89199d42313b8b"}
Oct 30 15:41:15.372880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.372823    5064 generic.go:356] "Generic (PLEG): container finished" podID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerID="54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406" exitCode=0
Oct 30 15:41:15.373063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.373047    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"
Oct 30 15:41:15.373183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.373167    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" event={"ID":"d13c2f8d-93e3-4bcc-b50a-f58954c035cd","Type":"ContainerDied","Data":"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"}
Oct 30 15:41:15.373233 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.373201    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7" event={"ID":"d13c2f8d-93e3-4bcc-b50a-f58954c035cd","Type":"ContainerDied","Data":"77741fabf5f4796ed23318c6eca80630d7baa57868a54ea48de25bb48f27626f"}
Oct 30 15:41:15.381405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.381378    5064 scope.go:122] "RemoveContainer" containerID="d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"
Oct 30 15:41:15.391713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.391687    5064 scope.go:122] "RemoveContainer" containerID="d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"
Oct 30 15:41:15.392086 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:15.392062    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72\": container with ID starting with d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72 not found: ID does not exist" containerID="d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"
Oct 30 15:41:15.392172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.392105    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72"} err="failed to get container status \"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72\": rpc error: code = NotFound desc = could not find container \"d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72\": container with ID starting with d0f3301ba8d6d59ae10363a91bdaf218c03546876878117f554c4d0bbb366f72 not found: ID does not exist"
Oct 30 15:41:15.392172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.392126    5064 scope.go:122] "RemoveContainer" containerID="54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"
Oct 30 15:41:15.393012 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.392988    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\" (UniqueName: \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\") pod \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") "
Oct 30 15:41:15.393128 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.393073    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\") pod \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\" (UID: \"603b328d-1d3e-42a1-a4cb-1fdab718ae2c\") "
Oct 30 15:41:15.393253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.393174    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\" (UniqueName: \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\") pod \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") "
Oct 30 15:41:15.393466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.393444    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume" pod "603b328d-1d3e-42a1-a4cb-1fdab718ae2c" (UID: "603b328d-1d3e-42a1-a4cb-1fdab718ae2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:15.393598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.393256    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\") pod \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\" (UID: \"d13c2f8d-93e3-4bcc-b50a-f58954c035cd\") "
Oct 30 15:41:15.394099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.394076    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:15.394200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.394139    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume" pod "d13c2f8d-93e3-4bcc-b50a-f58954c035cd" (UID: "d13c2f8d-93e3-4bcc-b50a-f58954c035cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:15.395728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.395705    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq" pod "d13c2f8d-93e3-4bcc-b50a-f58954c035cd" (UID: "d13c2f8d-93e3-4bcc-b50a-f58954c035cd"). InnerVolumeSpecName "kube-api-access-tnmtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:15.395892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.395876    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98" pod "603b328d-1d3e-42a1-a4cb-1fdab718ae2c" (UID: "603b328d-1d3e-42a1-a4cb-1fdab718ae2c"). InnerVolumeSpecName "kube-api-access-smz98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:15.402941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.402921    5064 scope.go:122] "RemoveContainer" containerID="54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"
Oct 30 15:41:15.403270 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:15.403251    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406\": container with ID starting with 54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406 not found: ID does not exist" containerID="54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"
Oct 30 15:41:15.403338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.403280    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406"} err="failed to get container status \"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406\": rpc error: code = NotFound desc = could not find container \"54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406\": container with ID starting with 54b33abcd0ee2cb0c4d684353b7691e44dff0d55dbcd4a7855c050512f694406 not found: ID does not exist"
Oct 30 15:41:15.465324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.465278    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 15:41:15.484191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.484147    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-357948-845c9476b7-fb97k"]
Oct 30 15:41:15.495065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.495031    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smz98\" (UniqueName: \"kubernetes.io/projected/603b328d-1d3e-42a1-a4cb-1fdab718ae2c-kube-api-access-smz98\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:15.495065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.495060    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmtq\" (UniqueName: \"kubernetes.io/projected/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-kube-api-access-tnmtq\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:15.495065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.495072    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13c2f8d-93e3-4bcc-b50a-f58954c035cd-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:15.730520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.730490    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:41:15.733362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.733334    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g"]
Oct 30 15:41:15.770331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.770289    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:41:15.773762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:15.773729    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-c57448747-8nqx7"]
Oct 30 15:41:16.081719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:16.081615    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-55885b5b6b-shk4g" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.198:9004: i/o timeout"
Oct 30 15:41:16.723042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:16.723005    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" path="/var/lib/kubelet/pods/377dfec6-1315-4dd7-81c3-32fbfdc9e6b9/volumes"
Oct 30 15:41:16.723450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:16.723436    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" path="/var/lib/kubelet/pods/603b328d-1d3e-42a1-a4cb-1fdab718ae2c/volumes"
Oct 30 15:41:16.723807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:16.723795    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" path="/var/lib/kubelet/pods/d13c2f8d-93e3-4bcc-b50a-f58954c035cd/volumes"
Oct 30 15:41:17.300036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.299969    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:41:17.322146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322112    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322171    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322213    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322235    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322269    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322311 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322300    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322345    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322397    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322458    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:17.322577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322479    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\") pod \"764bb822-4d34-40e2-aab0-b0b5933faf56\" (UID: \"764bb822-4d34-40e2-aab0-b0b5933faf56\") "
Oct 30 15:41:17.322805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322591    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:17.322805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322695    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:17.323007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.322959    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:17.323136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.323014    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:17.323246 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.323075    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.323353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.323333    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.323465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.323356    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764bb822-4d34-40e2-aab0-b0b5933faf56-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.324962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.324924    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:17.325182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.325159    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:17.325289 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.325272    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "kube-api-access-jxnb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:17.325352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.325335    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo" pod "764bb822-4d34-40e2-aab0-b0b5933faf56" (UID: "764bb822-4d34-40e2-aab0-b0b5933faf56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:17.394526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.394489    5064 generic.go:356] "Generic (PLEG): container finished" podID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerID="d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa" exitCode=0
Oct 30 15:41:17.394692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.394535    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerDied","Data":"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"}
Oct 30 15:41:17.394692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.394558    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"
Oct 30 15:41:17.394692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.394578    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8" event={"ID":"764bb822-4d34-40e2-aab0-b0b5933faf56","Type":"ContainerDied","Data":"da0ae635826e61026c5631479cc800b08c11cf7574c6b85ba206f75b21524b0f"}
Oct 30 15:41:17.394692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.394602    5064 scope.go:122] "RemoveContainer" containerID="d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"
Oct 30 15:41:17.405221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.405198    5064 scope.go:122] "RemoveContainer" containerID="e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"
Oct 30 15:41:17.417566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.417534    5064 scope.go:122] "RemoveContainer" containerID="d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"
Oct 30 15:41:17.417971 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:17.417944    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa\": container with ID starting with d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa not found: ID does not exist" containerID="d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"
Oct 30 15:41:17.418090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.417977    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa"} err="failed to get container status \"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa\": rpc error: code = NotFound desc = could not find container \"d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa\": container with ID starting with d179513732f9269bb3b0b809a4badaa83084de9e714b04f92db209ec7fdab0aa not found: ID does not exist"
Oct 30 15:41:17.418090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.417998    5064 scope.go:122] "RemoveContainer" containerID="e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"
Oct 30 15:41:17.418312 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:17.418292    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608\": container with ID starting with e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608 not found: ID does not exist" containerID="e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"
Oct 30 15:41:17.418376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.418317    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608"} err="failed to get container status \"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608\": rpc error: code = NotFound desc = could not find container \"e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608\": container with ID starting with e6048dc6be98fd7fb99cf22f5843af8cecd7103e40e04e1eed573bcb56e82608 not found: ID does not exist"
Oct 30 15:41:17.424463 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424431    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.424578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424468    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.424578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424484    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.424578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424498    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxnb7\" (UniqueName: \"kubernetes.io/projected/764bb822-4d34-40e2-aab0-b0b5933faf56-kube-api-access-jxnb7\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.424578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424512    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.424578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.424525    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764bb822-4d34-40e2-aab0-b0b5933faf56-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:17.451319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.451285    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:41:17.455375 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:17.455340    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-zzlr8"]
Oct 30 15:41:18.723460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:18.723426    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" path="/var/lib/kubelet/pods/764bb822-4d34-40e2-aab0-b0b5933faf56/volumes"
Oct 30 15:41:23.342171 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:23.342133    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:41:23.342696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:23.342421    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy" containerID="cri-o://6896a515ba4a5363a830dfe758bcd47ca6a8acc10f6666ea34e6cfcccba16bd5" gracePeriod=30
Oct 30 15:41:24.087907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:24.087862    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:41:24.088227 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:24.088132    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy" containerID="cri-o://e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754" gracePeriod=30
Oct 30 15:41:28.500813 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.500779    5064 generic.go:356] "Generic (PLEG): container finished" podID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerID="6896a515ba4a5363a830dfe758bcd47ca6a8acc10f6666ea34e6cfcccba16bd5" exitCode=0
Oct 30 15:41:28.501241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.500852    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" event={"ID":"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e","Type":"ContainerDied","Data":"6896a515ba4a5363a830dfe758bcd47ca6a8acc10f6666ea34e6cfcccba16bd5"}
Oct 30 15:41:28.629297 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.629263    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:41:28.770956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.770902    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.770971    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771045    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771158    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771214    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771264    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771290    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771341    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771354    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:28.771574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771385    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\") pod \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\" (UID: \"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e\") "
Oct 30 15:41:28.771574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771450    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:28.771574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771483    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:28.771908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771730    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:28.771908 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.771893    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:28.772136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.772112    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.772273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.772208    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.772273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.772226    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.772273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.772239    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.772273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.772252    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.774080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.774045    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:28.774205 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.774097    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "kube-api-access-hg499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:28.774331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.774313    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:28.774598 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.774332    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token" pod "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" (UID: "a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:28.873743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.873699    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.873743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.873733    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.873743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.873747    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:28.874021 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:28.873758    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg499\" (UniqueName: \"kubernetes.io/projected/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e-kube-api-access-hg499\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.351792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.351718    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:41:29.480313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480274    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480323    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480375    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480427    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480466    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480557    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480594    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480622    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:29.480771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480699    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.480771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480739    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\") pod \"30b59509-9b9a-4399-9f13-4453158404e4\" (UID: \"30b59509-9b9a-4399-9f13-4453158404e4\") "
Oct 30 15:41:29.481099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480953    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:29.481099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.480978    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:29.481099 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481014    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:29.481271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481100    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:29.481333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481315    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.481395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481334    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b59509-9b9a-4399-9f13-4453158404e4-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.481395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481348    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.481395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481363    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.481395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.481378    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.482876 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.482841    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "kube-api-access-kxdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:29.483003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.482877    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:29.483003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.482913    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:29.483084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.483017    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo" pod "30b59509-9b9a-4399-9f13-4453158404e4" (UID: "30b59509-9b9a-4399-9f13-4453158404e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:29.507812 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.507778    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"
Oct 30 15:41:29.508231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.507780    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp" event={"ID":"a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e","Type":"ContainerDied","Data":"88777d59fe241f7bd7a0df560754b089326ec2407d6e5767466406dfc4c08c69"}
Oct 30 15:41:29.508231 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.507897    5064 scope.go:122] "RemoveContainer" containerID="6896a515ba4a5363a830dfe758bcd47ca6a8acc10f6666ea34e6cfcccba16bd5"
Oct 30 15:41:29.509613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.509585    5064 generic.go:356] "Generic (PLEG): container finished" podID="30b59509-9b9a-4399-9f13-4453158404e4" containerID="e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754" exitCode=0
Oct 30 15:41:29.509730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.509674    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerDied","Data":"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"}
Oct 30 15:41:29.509730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.509722    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh" event={"ID":"30b59509-9b9a-4399-9f13-4453158404e4","Type":"ContainerDied","Data":"3fcd440645a8d5f797e753721e516733ae1cec90ac75c1af7c1358553a5b349c"}
Oct 30 15:41:29.509870 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.509792    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"
Oct 30 15:41:29.520760 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.520742    5064 scope.go:122] "RemoveContainer" containerID="e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"
Oct 30 15:41:29.530275 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.530256    5064 scope.go:122] "RemoveContainer" containerID="f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"
Oct 30 15:41:29.541565 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.541544    5064 scope.go:122] "RemoveContainer" containerID="e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"
Oct 30 15:41:29.541841 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:29.541807    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754\": container with ID starting with e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754 not found: ID does not exist" containerID="e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"
Oct 30 15:41:29.541907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.541856    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754"} err="failed to get container status \"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754\": rpc error: code = NotFound desc = could not find container \"e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754\": container with ID starting with e564b49504faed5985cb25b534fb5b87b7093952639d734485ffdcf2847f2754 not found: ID does not exist"
Oct 30 15:41:29.541907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.541875    5064 scope.go:122] "RemoveContainer" containerID="f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"
Oct 30 15:41:29.542134 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:29.542116    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe\": container with ID starting with f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe not found: ID does not exist" containerID="f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"
Oct 30 15:41:29.542183 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.542139    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe"} err="failed to get container status \"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe\": rpc error: code = NotFound desc = could not find container \"f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe\": container with ID starting with f41f5d17c8a9d4400d40b35a63e880d2db2472327e5e1db98c58526190ab4dfe not found: ID does not exist"
Oct 30 15:41:29.565110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.565078    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:41:29.573057 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.573027    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-01c89d-openshift-default-6d44dd799-fj9fh"]
Oct 30 15:41:29.582517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.582488    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b59509-9b9a-4399-9f13-4453158404e4-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.582652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.582518    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdrb\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-kube-api-access-kxdrb\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.582652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.582534    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b59509-9b9a-4399-9f13-4453158404e4-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.582652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.582546    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b59509-9b9a-4399-9f13-4453158404e4-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:29.610738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.610649    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:41:29.613444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:29.613418    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-fprzp"]
Oct 30 15:41:29.632313 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:29.632278    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:41:30.723879 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:30.723809    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b59509-9b9a-4399-9f13-4453158404e4" path="/var/lib/kubelet/pods/30b59509-9b9a-4399-9f13-4453158404e4/volumes"
Oct 30 15:41:30.724377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:30.724359    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" path="/var/lib/kubelet/pods/a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e/volumes"
Oct 30 15:41:35.810154 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:35.810113    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:41:35.810700 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:35.810420    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" podUID="25073711-e846-4908-93fb-2cf78b5bd4d5" containerName="maas-api" containerID="cri-o://96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4" gracePeriod=30
Oct 30 15:41:36.095485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.095452    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:41:36.259728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.256558    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") pod \"25073711-e846-4908-93fb-2cf78b5bd4d5\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") "
Oct 30 15:41:36.259728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.256652    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\") pod \"25073711-e846-4908-93fb-2cf78b5bd4d5\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") "
Oct 30 15:41:36.259728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.256770    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\") pod \"25073711-e846-4908-93fb-2cf78b5bd4d5\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") "
Oct 30 15:41:36.259728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.256974    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\") pod \"25073711-e846-4908-93fb-2cf78b5bd4d5\" (UID: \"25073711-e846-4908-93fb-2cf78b5bd4d5\") "
Oct 30 15:41:36.259728 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.258672    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca" pod "25073711-e846-4908-93fb-2cf78b5bd4d5" (UID: "25073711-e846-4908-93fb-2cf78b5bd4d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:36.264430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.264374    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs" pod "25073711-e846-4908-93fb-2cf78b5bd4d5" (UID: "25073711-e846-4908-93fb-2cf78b5bd4d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:36.266235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.266201    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss" pod "25073711-e846-4908-93fb-2cf78b5bd4d5" (UID: "25073711-e846-4908-93fb-2cf78b5bd4d5"). InnerVolumeSpecName "kube-api-access-564ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:36.292408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.292366    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls" pod "25073711-e846-4908-93fb-2cf78b5bd4d5" (UID: "25073711-e846-4908-93fb-2cf78b5bd4d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:36.359547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.359458    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:36.359547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.359492    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/25073711-e846-4908-93fb-2cf78b5bd4d5-kube-api-access-564ss\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:36.359547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.359505    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073711-e846-4908-93fb-2cf78b5bd4d5-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:36.359547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.359518    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073711-e846-4908-93fb-2cf78b5bd4d5-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:36.552415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.552374    5064 generic.go:356] "Generic (PLEG): container finished" podID="25073711-e846-4908-93fb-2cf78b5bd4d5" containerID="96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4" exitCode=0
Oct 30 15:41:36.552594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.552426    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"
Oct 30 15:41:36.552594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.552459    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" event={"ID":"25073711-e846-4908-93fb-2cf78b5bd4d5","Type":"ContainerDied","Data":"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"}
Oct 30 15:41:36.552594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.552523    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck" event={"ID":"25073711-e846-4908-93fb-2cf78b5bd4d5","Type":"ContainerDied","Data":"de3f09be5a402f5bac2faff167febf4a7c570d23c0b9d8a206da92909adfc4c6"}
Oct 30 15:41:36.552594 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.552550    5064 scope.go:122] "RemoveContainer" containerID="96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"
Oct 30 15:41:36.563276 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.563255    5064 scope.go:122] "RemoveContainer" containerID="96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"
Oct 30 15:41:36.563545 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:36.563526    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4\": container with ID starting with 96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4 not found: ID does not exist" containerID="96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"
Oct 30 15:41:36.563600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.563555    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4"} err="failed to get container status \"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4\": rpc error: code = NotFound desc = could not find container \"96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4\": container with ID starting with 96a9608d8cb3460c475ce6c70f671118398f89b069ae21f9eaf60a3cc05624e4 not found: ID does not exist"
Oct 30 15:41:36.607666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.607622    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:41:36.621263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.621225    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b236a6-75bf4ffc54-xlsck"]
Oct 30 15:41:36.722446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:36.722410    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25073711-e846-4908-93fb-2cf78b5bd4d5" path="/var/lib/kubelet/pods/25073711-e846-4908-93fb-2cf78b5bd4d5/volumes"
Oct 30 15:41:40.122304 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.122258    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 15:41:40.122792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.122479    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" podUID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" containerName="maas-api" containerID="cri-o://cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3" gracePeriod=30
Oct 30 15:41:40.399548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.399519    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 15:41:40.507351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.507315    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\" (UniqueName: \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\") pod \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") "
Oct 30 15:41:40.507550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.507379    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\") pod \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") "
Oct 30 15:41:40.507550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.507405    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\") pod \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") "
Oct 30 15:41:40.507550 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.507445    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\") pod \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\" (UID: \"a07ec3a0-dfc5-4756-870c-9199ce37a0d0\") "
Oct 30 15:41:40.507923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.507894    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca" pod "a07ec3a0-dfc5-4756-870c-9199ce37a0d0" (UID: "a07ec3a0-dfc5-4756-870c-9199ce37a0d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:40.509646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.509615    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls" pod "a07ec3a0-dfc5-4756-870c-9199ce37a0d0" (UID: "a07ec3a0-dfc5-4756-870c-9199ce37a0d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:40.510181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.510156    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv" pod "a07ec3a0-dfc5-4756-870c-9199ce37a0d0" (UID: "a07ec3a0-dfc5-4756-870c-9199ce37a0d0"). InnerVolumeSpecName "kube-api-access-mnfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:40.510257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.510171    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs" pod "a07ec3a0-dfc5-4756-870c-9199ce37a0d0" (UID: "a07ec3a0-dfc5-4756-870c-9199ce37a0d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:41:40.575537 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.575495    5064 generic.go:356] "Generic (PLEG): container finished" podID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" containerID="cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3" exitCode=0
Oct 30 15:41:40.575751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.575625    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" event={"ID":"a07ec3a0-dfc5-4756-870c-9199ce37a0d0","Type":"ContainerDied","Data":"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"}
Oct 30 15:41:40.575751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.575677    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw" event={"ID":"a07ec3a0-dfc5-4756-870c-9199ce37a0d0","Type":"ContainerDied","Data":"1993449582af4f63871f530437c19cd906ebca6d03c55ec25206d7da9eeb2992"}
Oct 30 15:41:40.575751 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.575708    5064 scope.go:122] "RemoveContainer" containerID="cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"
Oct 30 15:41:40.575955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.575896    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"
Oct 30 15:41:40.598711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.598684    5064 scope.go:122] "RemoveContainer" containerID="cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"
Oct 30 15:41:40.599183 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:40.599155    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3\": container with ID starting with cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3 not found: ID does not exist" containerID="cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"
Oct 30 15:41:40.599296 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.599192    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3"} err="failed to get container status \"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3\": rpc error: code = NotFound desc = could not find container \"cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3\": container with ID starting with cf667f4b11da61c512260158683fb315a91ea5d758096edac93589c947e983e3 not found: ID does not exist"
Oct 30 15:41:40.608890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.608865    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:40.608890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.608889    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:40.609056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.608898    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:40.609056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.608909    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfcv\" (UniqueName: \"kubernetes.io/projected/a07ec3a0-dfc5-4756-870c-9199ce37a0d0-kube-api-access-mnfcv\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:40.639901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.639786    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 15:41:40.643553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.643519    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e64e05-8677d47d8f-9hkqw"]
Oct 30 15:41:40.722340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:40.722306    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" path="/var/lib/kubelet/pods/a07ec3a0-dfc5-4756-870c-9199ce37a0d0/volumes"
Oct 30 15:41:44.775417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:44.775377    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:41:44.776029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:44.775610    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy" containerID="cri-o://8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034" gracePeriod=30
Oct 30 15:41:48.067028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:48.066983    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:41:48.067906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:48.067867    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy" containerID="cri-o://6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427" gracePeriod=30
Oct 30 15:41:50.045796 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.045735    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:41:50.123643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123600    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.123872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123659    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.123872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123687    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.123872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123717    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.123872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123742    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.123872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123859    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.124197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.123904    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.124197 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124129    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.124313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124203    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\") pod \"47c696c5-8483-48ba-8f51-afafcfb43d9f\" (UID: \"47c696c5-8483-48ba-8f51-afafcfb43d9f\") "
Oct 30 15:41:50.124313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124276    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:50.124432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124398    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:50.124717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124487    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:50.124717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124683    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:50.124996 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.124803    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:50.125158 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.125132    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c696c5-8483-48ba-8f51-afafcfb43d9f-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.125267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.125159    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.125267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.125176    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.125267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.125190    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.125267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.125203    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.127001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.126822    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "kube-api-access-xn862". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:50.127001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.126968    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:50.127001 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.126979    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:50.127803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.127774    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo" pod "47c696c5-8483-48ba-8f51-afafcfb43d9f" (UID: "47c696c5-8483-48ba-8f51-afafcfb43d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:50.226331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.226288    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.226331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.226330    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn862\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-kube-api-access-xn862\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.226549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.226346    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.226549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.226360    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c696c5-8483-48ba-8f51-afafcfb43d9f-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:50.631138 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.631102    5064 generic.go:356] "Generic (PLEG): container finished" podID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerID="8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034" exitCode=0
Oct 30 15:41:50.631353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.631166    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"
Oct 30 15:41:50.631353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.631191    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" event={"ID":"47c696c5-8483-48ba-8f51-afafcfb43d9f","Type":"ContainerDied","Data":"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"}
Oct 30 15:41:50.631353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.631284    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl" event={"ID":"47c696c5-8483-48ba-8f51-afafcfb43d9f","Type":"ContainerDied","Data":"0dba2cc267da1d7017e8dc165d0fd9fb59578c6e0523135366f5d814501a8f37"}
Oct 30 15:41:50.631353 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.631314    5064 scope.go:122] "RemoveContainer" containerID="8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"
Oct 30 15:41:50.642141 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.642120    5064 scope.go:122] "RemoveContainer" containerID="8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"
Oct 30 15:41:50.642461 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:50.642441    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034\": container with ID starting with 8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034 not found: ID does not exist" containerID="8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"
Oct 30 15:41:50.642548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.642475    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034"} err="failed to get container status \"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034\": rpc error: code = NotFound desc = could not find container \"8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034\": container with ID starting with 8c8d659592b93b5a9f90a0a647e5d7f4c9eb052a4c0140777877bf9eb6ee1034 not found: ID does not exist"
Oct 30 15:41:50.714008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.713964    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:41:50.726039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:50.726007    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-5dpsl"]
Oct 30 15:41:52.722473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:52.722442    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" path="/var/lib/kubelet/pods/47c696c5-8483-48ba-8f51-afafcfb43d9f/volumes"
Oct 30 15:41:53.343719 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.343653    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:41:53.456062 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456026    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456085    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456149    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456216    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456297    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456326    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456362    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456793 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456761    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456793 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456392    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:53.456968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456794    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\") pod \"c861cffd-0939-49dd-8ea8-9f703bb9c942\" (UID: \"c861cffd-0939-49dd-8ea8-9f703bb9c942\") "
Oct 30 15:41:53.456968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456404    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:53.456968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456662    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:53.456968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.456710    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:41:53.457194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457008    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:53.457261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457229    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.457261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457243    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861cffd-0939-49dd-8ea8-9f703bb9c942-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.457261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457254    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.457435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457268    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.457435 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.457280    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.458678 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.458651    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:41:53.458773 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.458693    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:53.459040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.459020    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "kube-api-access-q947h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:41:53.459113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.459071    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy" pod "c861cffd-0939-49dd-8ea8-9f703bb9c942" (UID: "c861cffd-0939-49dd-8ea8-9f703bb9c942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:41:53.558334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.558295    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.558334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.558327    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q947h\" (UniqueName: \"kubernetes.io/projected/c861cffd-0939-49dd-8ea8-9f703bb9c942-kube-api-access-q947h\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.558334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.558339    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.558584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.558349    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861cffd-0939-49dd-8ea8-9f703bb9c942-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:41:53.649464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.649423    5064 generic.go:356] "Generic (PLEG): container finished" podID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerID="6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427" exitCode=0
Oct 30 15:41:53.649624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.649485    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"
Oct 30 15:41:53.649624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.649506    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" event={"ID":"c861cffd-0939-49dd-8ea8-9f703bb9c942","Type":"ContainerDied","Data":"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"}
Oct 30 15:41:53.649624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.649562    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s" event={"ID":"c861cffd-0939-49dd-8ea8-9f703bb9c942","Type":"ContainerDied","Data":"915e148d91e8e3cd76f7fb5befd7f98186d7f1ec12539ac5495ad6d041f46cac"}
Oct 30 15:41:53.649624 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.649588    5064 scope.go:122] "RemoveContainer" containerID="6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"
Oct 30 15:41:53.660211 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.660186    5064 scope.go:122] "RemoveContainer" containerID="6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"
Oct 30 15:41:53.660571 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:53.660542    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427\": container with ID starting with 6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427 not found: ID does not exist" containerID="6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"
Oct 30 15:41:53.660636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.660584    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427"} err="failed to get container status \"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427\": rpc error: code = NotFound desc = could not find container \"6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427\": container with ID starting with 6c5c6386e83f5321b344f3cc4279094b351d208db962845659b229c722085427 not found: ID does not exist"
Oct 30 15:41:53.706113 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.706077    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:41:53.719634 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:53.719600    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-ztt6s"]
Oct 30 15:41:54.721182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:41:54.721145    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" path="/var/lib/kubelet/pods/c861cffd-0939-49dd-8ea8-9f703bb9c942/volumes"
Oct 30 15:41:59.629193 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:41:59.629157    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:42:04.810080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:04.810043    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:42:04.810473 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:04.810245    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy" containerID="cri-o://9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687" gracePeriod=30
Oct 30 15:42:10.077630 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.077563    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:42:10.240131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240091    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240148    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240173    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240229    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240271    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240309 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240301    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240398    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240427    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240455    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\") pod \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\" (UID: \"6a745e2d-c9e6-48fc-af52-8db0d737a05b\") "
Oct 30 15:42:10.240655 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240642    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:10.240915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240658    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:10.240991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240884    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:10.240991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240979    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.241120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.240995    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:42:10.241120 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.241002    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.241236 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.241195    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:10.242768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.242744    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:10.242768 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.242754    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:10.242999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.242981    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "kube-api-access-5frkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:10.243071 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.243055    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo" pod "6a745e2d-c9e6-48fc-af52-8db0d737a05b" (UID: "6a745e2d-c9e6-48fc-af52-8db0d737a05b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:42:10.299663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.299617    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:42:10.333663 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.333567    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:42:10.342582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342550    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342580    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342595    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342606    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342617    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342629    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5frkd\" (UniqueName: \"kubernetes.io/projected/6a745e2d-c9e6-48fc-af52-8db0d737a05b-kube-api-access-5frkd\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.342810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.342642    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a745e2d-c9e6-48fc-af52-8db0d737a05b-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:10.739957 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.739922    5064 generic.go:356] "Generic (PLEG): container finished" podID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerID="9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687" exitCode=0
Oct 30 15:42:10.740102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.739980    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"
Oct 30 15:42:10.740102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.740010    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerDied","Data":"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"}
Oct 30 15:42:10.740102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.740065    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm" event={"ID":"6a745e2d-c9e6-48fc-af52-8db0d737a05b","Type":"ContainerDied","Data":"696f9cc9a531383daef2d251034798cfda4ff3f735c6fbfc77fa9493f2999dd1"}
Oct 30 15:42:10.740102 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.740088    5064 scope.go:122] "RemoveContainer" containerID="9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"
Oct 30 15:42:10.750536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.750514    5064 scope.go:122] "RemoveContainer" containerID="e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"
Oct 30 15:42:10.759903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.759881    5064 scope.go:122] "RemoveContainer" containerID="9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"
Oct 30 15:42:10.760221 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:10.760198    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687\": container with ID starting with 9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687 not found: ID does not exist" containerID="9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"
Oct 30 15:42:10.760274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.760232    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687"} err="failed to get container status \"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687\": rpc error: code = NotFound desc = could not find container \"9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687\": container with ID starting with 9162d8cf9b68ef47619fb2294321ffa8a74d8ac661ed3581c940ce32c08bd687 not found: ID does not exist"
Oct 30 15:42:10.760274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.760250    5064 scope.go:122] "RemoveContainer" containerID="e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"
Oct 30 15:42:10.760505 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:10.760486    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40\": container with ID starting with e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40 not found: ID does not exist" containerID="e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"
Oct 30 15:42:10.760556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.760512    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40"} err="failed to get container status \"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40\": rpc error: code = NotFound desc = could not find container \"e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40\": container with ID starting with e0ea0cbbee784fb3515911b76604b6310cc1875e746d6ad4fea5463484726b40 not found: ID does not exist"
Oct 30 15:42:10.797952 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.797856    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:42:10.811201 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:10.811165    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-dtgmm"]
Oct 30 15:42:12.721805 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:12.721765    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" path="/var/lib/kubelet/pods/6a745e2d-c9e6-48fc-af52-8db0d737a05b/volumes"
Oct 30 15:42:25.320708 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.320653    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main" containerID="cri-o://1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102" gracePeriod=45
Oct 30 15:42:25.354660 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.354617    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" containerID="cri-o://9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1" gracePeriod=45
Oct 30 15:42:25.359048 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.359017    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": EOF"
Oct 30 15:42:25.626317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.625565    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:42:25.631290 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.631261    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:42:25.702652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702612    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.702868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702699    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\" (UniqueName: \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.702868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702737    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.702868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702777    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.702868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702804    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702880    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702913    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702929    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.702950    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703032    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\" (UniqueName: \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703082    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\" (UniqueName: \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.703136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703107    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.703493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703134    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\") pod \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\" (UID: \"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18\") "
Oct 30 15:42:25.703493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703234    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\" (UniqueName: \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\") pod \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\" (UID: \"183f88c0-d9d3-4195-a7a2-a0ed402f2265\") "
Oct 30 15:42:25.703493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703402    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.703733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703708    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.703940 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703901    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.704117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703947    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.704117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.703963    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.704117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.704021    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.704117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.704072    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.704425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.704403    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.705923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.705890    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.706098 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.705914    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:25.706340 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.706321    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:25.707029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.706995    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:25.707129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.707007    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2" pod "183f88c0-d9d3-4195-a7a2-a0ed402f2265" (UID: "183f88c0-d9d3-4195-a7a2-a0ed402f2265"). InnerVolumeSpecName "kube-api-access-94lm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:25.708053 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.708036    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt" pod "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" (UID: "8002e35d-46ef-4c8f-a3fa-1adbaee4bf18"). InnerVolumeSpecName "kube-api-access-pk6gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:25.804980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.804937    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.804980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.804977    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.804980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.804986    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.805243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.804993    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f88c0-d9d3-4195-a7a2-a0ed402f2265-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.805243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.805003    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94lm2\" (UniqueName: \"kubernetes.io/projected/183f88c0-d9d3-4195-a7a2-a0ed402f2265-kube-api-access-94lm2\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.805243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.805013    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6gt\" (UniqueName: \"kubernetes.io/projected/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-kube-api-access-pk6gt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.805243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.805021    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.805243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.805030    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f88c0-d9d3-4195-a7a2-a0ed402f2265-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:25.826271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.826231    5064 generic.go:356] "Generic (PLEG): container finished" podID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerID="9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1" exitCode=0
Oct 30 15:42:25.826472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.826298    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"
Oct 30 15:42:25.826472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.826306    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" event={"ID":"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18","Type":"ContainerDied","Data":"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"}
Oct 30 15:42:25.826472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.826344    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf" event={"ID":"8002e35d-46ef-4c8f-a3fa-1adbaee4bf18","Type":"ContainerDied","Data":"338bea5f6cfc8f94e315fdf564e9e2ac6129bea3cee4099fdbdf7bef897c02d8"}
Oct 30 15:42:25.826472 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.826362    5064 scope.go:122] "RemoveContainer" containerID="9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"
Oct 30 15:42:25.827808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.827785    5064 generic.go:356] "Generic (PLEG): container finished" podID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerID="1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102" exitCode=0
Oct 30 15:42:25.827968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.827815    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" event={"ID":"183f88c0-d9d3-4195-a7a2-a0ed402f2265","Type":"ContainerDied","Data":"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"}
Oct 30 15:42:25.827968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.827870    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4" event={"ID":"183f88c0-d9d3-4195-a7a2-a0ed402f2265","Type":"ContainerDied","Data":"348b90de2f75cb3171d702fcc175ec15f2656cfd409ffad506e19e40a4fea527"}
Oct 30 15:42:25.827968 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.827889    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"
Oct 30 15:42:25.840039 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.840015    5064 scope.go:122] "RemoveContainer" containerID="9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"
Oct 30 15:42:25.840635 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:25.840445    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1\": container with ID starting with 9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1 not found: ID does not exist" containerID="9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"
Oct 30 15:42:25.840635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.840484    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1"} err="failed to get container status \"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1\": rpc error: code = NotFound desc = could not find container \"9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1\": container with ID starting with 9334d3e90e15e5f85c107b09093fe174e60add13fbfe5493ded6554cea789cb1 not found: ID does not exist"
Oct 30 15:42:25.840635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.840506    5064 scope.go:122] "RemoveContainer" containerID="1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"
Oct 30 15:42:25.850990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.850972    5064 scope.go:122] "RemoveContainer" containerID="1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"
Oct 30 15:42:25.851274 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:25.851256    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102\": container with ID starting with 1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102 not found: ID does not exist" containerID="1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"
Oct 30 15:42:25.851332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.851282    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102"} err="failed to get container status \"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102\": rpc error: code = NotFound desc = could not find container \"1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102\": container with ID starting with 1a51f5c8d1f6ef200232ca690ade3afc426c4b8d580b502538a7fc4972c72102 not found: ID does not exist"
Oct 30 15:42:25.882213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.882178    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:42:25.889501 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.889467    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/premium-w1-83ff4e-kserve-5bd5fbb89f-dcgzf"]
Oct 30 15:42:25.926993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.926960    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:42:25.934823 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:25.934788    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-2-w1-83ff4e-kserve-bb55f4885-nqwj4"]
Oct 30 15:42:26.722696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:26.722664    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" path="/var/lib/kubelet/pods/183f88c0-d9d3-4195-a7a2-a0ed402f2265/volumes"
Oct 30 15:42:26.723116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:26.723102    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" path="/var/lib/kubelet/pods/8002e35d-46ef-4c8f-a3fa-1adbaee4bf18/volumes"
Oct 30 15:42:29.630511 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:29.630468    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:42:31.148562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.148525    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 15:42:31.149250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.148799    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" podUID="bb5d297a-1d44-4c2e-ba81-7578644270e7" containerName="maas-api" containerID="cri-o://b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8" gracePeriod=30
Oct 30 15:42:31.399786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.399712    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:42:31.407715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.407689    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:42:31.438926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.438891    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 15:42:31.559131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.559093    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\" (UniqueName: \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\") pod \"bb5d297a-1d44-4c2e-ba81-7578644270e7\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") "
Oct 30 15:42:31.559357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.559140    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\") pod \"bb5d297a-1d44-4c2e-ba81-7578644270e7\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") "
Oct 30 15:42:31.559357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.559172    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\") pod \"bb5d297a-1d44-4c2e-ba81-7578644270e7\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") "
Oct 30 15:42:31.559357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.559278    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\") pod \"bb5d297a-1d44-4c2e-ba81-7578644270e7\" (UID: \"bb5d297a-1d44-4c2e-ba81-7578644270e7\") "
Oct 30 15:42:31.559621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.559590    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca" pod "bb5d297a-1d44-4c2e-ba81-7578644270e7" (UID: "bb5d297a-1d44-4c2e-ba81-7578644270e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:42:31.561514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.561480    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt" pod "bb5d297a-1d44-4c2e-ba81-7578644270e7" (UID: "bb5d297a-1d44-4c2e-ba81-7578644270e7"). InnerVolumeSpecName "kube-api-access-ktqjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:31.561615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.561513    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls" pod "bb5d297a-1d44-4c2e-ba81-7578644270e7" (UID: "bb5d297a-1d44-4c2e-ba81-7578644270e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:31.561615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.561536    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs" pod "bb5d297a-1d44-4c2e-ba81-7578644270e7" (UID: "bb5d297a-1d44-4c2e-ba81-7578644270e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:31.660824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.660735    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqjt\" (UniqueName: \"kubernetes.io/projected/bb5d297a-1d44-4c2e-ba81-7578644270e7-kube-api-access-ktqjt\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:31.660824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.660769    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297a-1d44-4c2e-ba81-7578644270e7-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:31.660824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.660780    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:31.660824 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.660789    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297a-1d44-4c2e-ba81-7578644270e7-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:31.863493 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.863461    5064 generic.go:356] "Generic (PLEG): container finished" podID="bb5d297a-1d44-4c2e-ba81-7578644270e7" containerID="b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8" exitCode=0
Oct 30 15:42:31.863699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.863509    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"
Oct 30 15:42:31.863699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.863533    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" event={"ID":"bb5d297a-1d44-4c2e-ba81-7578644270e7","Type":"ContainerDied","Data":"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"}
Oct 30 15:42:31.863699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.863601    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht" event={"ID":"bb5d297a-1d44-4c2e-ba81-7578644270e7","Type":"ContainerDied","Data":"e2f76635f9241be4293e6663785325b22c30a7eb01bb99dc7b45631e074505fc"}
Oct 30 15:42:31.863699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.863635    5064 scope.go:122] "RemoveContainer" containerID="b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"
Oct 30 15:42:31.874136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.874114    5064 scope.go:122] "RemoveContainer" containerID="b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"
Oct 30 15:42:31.874434 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:31.874415    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8\": container with ID starting with b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8 not found: ID does not exist" containerID="b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"
Oct 30 15:42:31.874496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.874442    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8"} err="failed to get container status \"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8\": rpc error: code = NotFound desc = could not find container \"b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8\": container with ID starting with b86fad8bc1e99521176f18c584339de9bac2a5f4b70a6fa5857eed1877a6bcc8 not found: ID does not exist"
Oct 30 15:42:31.929797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.929705    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 15:42:31.936941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:31.936903    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-83ff4e-86845574ff-4rvht"]
Oct 30 15:42:32.721439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:32.721405    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5d297a-1d44-4c2e-ba81-7578644270e7" path="/var/lib/kubelet/pods/bb5d297a-1d44-4c2e-ba81-7578644270e7/volumes"
Oct 30 15:42:39.133946 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:39.133903    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 15:42:39.134452 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:39.134170    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" containerID="cri-o://baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10" gracePeriod=30
Oct 30 15:42:42.461427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:42.461385    5064 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:42:42.461888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:42.461446    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:42:44.053520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.053477    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 15:42:44.053973 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.053706    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68588d554c-9fpwf" podUID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" containerName="manager" containerID="cri-o://8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7" gracePeriod=30
Oct 30 15:42:44.388498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.388469    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 15:42:44.436695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.435446    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:42:44.493774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.493737    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\") pod \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") "
Oct 30 15:42:44.493981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.493809    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\") pod \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") "
Oct 30 15:42:44.493981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.493867    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\" (UniqueName: \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\") pod \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\" (UID: \"46d3e086-bc80-43e2-aba5-190c8ff3cc4a\") "
Oct 30 15:42:44.497455 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.497408    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm" pod "46d3e086-bc80-43e2-aba5-190c8ff3cc4a" (UID: "46d3e086-bc80-43e2-aba5-190c8ff3cc4a"). InnerVolumeSpecName "kube-api-access-7pjqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:44.501111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.501067    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs" pod "46d3e086-bc80-43e2-aba5-190c8ff3cc4a" (UID: "46d3e086-bc80-43e2-aba5-190c8ff3cc4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:44.502560 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.502528    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert" pod "46d3e086-bc80-43e2-aba5-190c8ff3cc4a" (UID: "46d3e086-bc80-43e2-aba5-190c8ff3cc4a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:42:44.595056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595015    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595063    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595121    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595292    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595346    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595512 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595486    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595534    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595541    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:42:44.595586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595567    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595650    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\") pod \"cec3b842-e8c3-42fe-a24f-0c3def05b372\" (UID: \"cec3b842-e8c3-42fe-a24f-0c3def05b372\") "
Oct 30 15:42:44.595874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595665    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:44.595874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.595802    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:44.596055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596039    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:44.596116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596054    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:44.596178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596110    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.596178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596130    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.596178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596143    5064 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-webhook-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.596178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596157    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec3b842-e8c3-42fe-a24f-0c3def05b372-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.596178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596170    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.596459 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.596183    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjqm\" (UniqueName: \"kubernetes.io/projected/46d3e086-bc80-43e2-aba5-190c8ff3cc4a-kube-api-access-7pjqm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.597556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.597531    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:42:44.597670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.597629    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:44.597780 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.597760    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:42:44.598332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.598314    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh" pod "cec3b842-e8c3-42fe-a24f-0c3def05b372" (UID: "cec3b842-e8c3-42fe-a24f-0c3def05b372"). InnerVolumeSpecName "kube-api-access-spmhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:42:44.696901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696867    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.696901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696902    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.697180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696916    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec3b842-e8c3-42fe-a24f-0c3def05b372-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.697180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696930    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.697180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696950    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spmhh\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-kube-api-access-spmhh\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.697180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.696965    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec3b842-e8c3-42fe-a24f-0c3def05b372-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:42:44.936137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.936104    5064 generic.go:356] "Generic (PLEG): container finished" podID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerID="baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10" exitCode=0
Oct 30 15:42:44.936334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.936152    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"
Oct 30 15:42:44.936334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.936196    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerDied","Data":"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"}
Oct 30 15:42:44.936334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.936264    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv" event={"ID":"cec3b842-e8c3-42fe-a24f-0c3def05b372","Type":"ContainerDied","Data":"908b605dae501251bf417e4b394c6c6b4d778a35e5a4f2243da743018c1a41b3"}
Oct 30 15:42:44.936334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.936284    5064 scope.go:122] "RemoveContainer" containerID="baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"
Oct 30 15:42:44.937684 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.937650    5064 generic.go:356] "Generic (PLEG): container finished" podID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" containerID="8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7" exitCode=0
Oct 30 15:42:44.937785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.937688    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9fpwf"
Oct 30 15:42:44.937785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.937734    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9fpwf" event={"ID":"46d3e086-bc80-43e2-aba5-190c8ff3cc4a","Type":"ContainerDied","Data":"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"}
Oct 30 15:42:44.937906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.937795    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9fpwf" event={"ID":"46d3e086-bc80-43e2-aba5-190c8ff3cc4a","Type":"ContainerDied","Data":"5127e828f674ba1f3a02e7a98581eb2ef779cf26df8dba888a4a2e2a4e2254cf"}
Oct 30 15:42:44.945943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.945919    5064 scope.go:122] "RemoveContainer" containerID="e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"
Oct 30 15:42:44.956916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.956893    5064 scope.go:122] "RemoveContainer" containerID="baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"
Oct 30 15:42:44.957275 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:44.957247    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10\": container with ID starting with baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10 not found: ID does not exist" containerID="baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"
Oct 30 15:42:44.957382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.957283    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10"} err="failed to get container status \"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10\": rpc error: code = NotFound desc = could not find container \"baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10\": container with ID starting with baf7ef3b793db22fff53a3aac0371f10552f7a6bd71e324aacc48b15f240aa10 not found: ID does not exist"
Oct 30 15:42:44.957382 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.957302    5064 scope.go:122] "RemoveContainer" containerID="e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"
Oct 30 15:42:44.957641 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:44.957624    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e\": container with ID starting with e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e not found: ID does not exist" containerID="e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"
Oct 30 15:42:44.957690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.957646    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e"} err="failed to get container status \"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e\": rpc error: code = NotFound desc = could not find container \"e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e\": container with ID starting with e4f048f79441870543443273d782b5e15305d9bd00728cbb474e613bc65d350e not found: ID does not exist"
Oct 30 15:42:44.957690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.957661    5064 scope.go:122] "RemoveContainer" containerID="8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"
Oct 30 15:42:44.967447 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.967424    5064 scope.go:122] "RemoveContainer" containerID="8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"
Oct 30 15:42:44.967733 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:44.967712    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7\": container with ID starting with 8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7 not found: ID does not exist" containerID="8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"
Oct 30 15:42:44.967803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.967745    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7"} err="failed to get container status \"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7\": rpc error: code = NotFound desc = could not find container \"8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7\": container with ID starting with 8eec7f57df60fa5a59a8da04fefc12f34d534ec5836ab12673e43cb2375b4ec7 not found: ID does not exist"
Oct 30 15:42:44.997535 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:44.997505    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 15:42:45.011146 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:45.011120    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-lnzgv"]
Oct 30 15:42:45.042408 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:45.042379    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 15:42:45.046303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:45.046274    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68588d554c-9fpwf"]
Oct 30 15:42:46.721928 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.721893    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" path="/var/lib/kubelet/pods/46d3e086-bc80-43e2-aba5-190c8ff3cc4a/volumes"
Oct 30 15:42:46.722443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.722352    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" path="/var/lib/kubelet/pods/cec3b842-e8c3-42fe-a24f-0c3def05b372/volumes"
Oct 30 15:42:46.760213 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760175    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:42:46.760639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760624    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy"
Oct 30 15:42:46.760691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760641    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy"
Oct 30 15:42:46.760691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760665    5064 cpu_manager.go:414] "containerMap: removing container" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.760691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760673    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.760691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760682    5064 cpu_manager.go:414] "containerMap: removing container" podUID="25073711-e846-4908-93fb-2cf78b5bd4d5" containerName="maas-api"
Oct 30 15:42:46.760691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760687    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25073711-e846-4908-93fb-2cf78b5bd4d5" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760695    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerName="payload-pre-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760701    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerName="payload-pre-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760708    5064 cpu_manager.go:414] "containerMap: removing container" podUID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760713    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760723    5064 cpu_manager.go:414] "containerMap: removing container" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760729    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760742    5064 cpu_manager.go:414] "containerMap: removing container" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760746    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760753    5064 cpu_manager.go:414] "containerMap: removing container" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760758    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760766    5064 cpu_manager.go:414] "containerMap: removing container" podUID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760771    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760779    5064 cpu_manager.go:414] "containerMap: removing container" podUID="55005f68-0a05-4bc9-aabf-fcd422046bd6" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760783    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55005f68-0a05-4bc9-aabf-fcd422046bd6" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760790    5064 cpu_manager.go:414] "containerMap: removing container" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760795    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760802    5064 cpu_manager.go:414] "containerMap: removing container" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760807    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760816    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760823    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760839    5064 cpu_manager.go:414] "containerMap: removing container" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760844    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerName="maas-api"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760852    5064 cpu_manager.go:414] "containerMap: removing container" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760856    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760863    5064 cpu_manager.go:414] "containerMap: removing container" podUID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" containerName="manager"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760868    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" containerName="manager"
Oct 30 15:42:46.760864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760874    5064 cpu_manager.go:414] "containerMap: removing container" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760879    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760887    5064 cpu_manager.go:414] "containerMap: removing container" podUID="de1735e4-781b-437e-8b5b-3b1ecb784e13" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760894    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1735e4-781b-437e-8b5b-3b1ecb784e13" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760906    5064 cpu_manager.go:414] "containerMap: removing container" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760911    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760917    5064 cpu_manager.go:414] "containerMap: removing container" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760922    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760929    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760933    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760946    5064 cpu_manager.go:414] "containerMap: removing container" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760951    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760958    5064 cpu_manager.go:414] "containerMap: removing container" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760962    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760974    5064 cpu_manager.go:414] "containerMap: removing container" podUID="9ba81452-32ec-4672-9892-2dbbabe73285" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760981    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba81452-32ec-4672-9892-2dbbabe73285" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760988    5064 cpu_manager.go:414] "containerMap: removing container" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.760992    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761000    5064 cpu_manager.go:414] "containerMap: removing container" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761004    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761011    5064 cpu_manager.go:414] "containerMap: removing container" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761015    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761021    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761025    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761031    5064 cpu_manager.go:414] "containerMap: removing container" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761035    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761051    5064 cpu_manager.go:414] "containerMap: removing container" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerName="payload-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761057    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerName="payload-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761066    5064 cpu_manager.go:414] "containerMap: removing container" podUID="aabf3fae-02b3-494e-86e0-edf307a702a8" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761070    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aabf3fae-02b3-494e-86e0-edf307a702a8" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761076    5064 cpu_manager.go:414] "containerMap: removing container" podUID="bb5d297a-1d44-4c2e-ba81-7578644270e7" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761080    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5d297a-1d44-4c2e-ba81-7578644270e7" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761154    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761166    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="603b328d-1d3e-42a1-a4cb-1fdab718ae2c" containerName="payload-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761174    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="aabf3fae-02b3-494e-86e0-edf307a702a8" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761179    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5d297a-1d44-4c2e-ba81-7578644270e7" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761185    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e5cde7-d31e-4efb-a40e-5f91ffa0e54e" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761193    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761198    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="25073711-e846-4908-93fb-2cf78b5bd4d5" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761209    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13c2f8d-93e3-4bcc-b50a-f58954c035cd" containerName="payload-pre-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761217    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ec3a0-dfc5-4756-870c-9199ce37a0d0" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761223    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761227    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761232    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a745e2d-c9e6-48fc-af52-8db0d737a05b" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761237    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="377dfec6-1315-4dd7-81c3-32fbfdc9e6b9" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761242    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="55005f68-0a05-4bc9-aabf-fcd422046bd6" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761246    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad20a01-4a33-461c-9919-fbcbda8f6158" containerName="main"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761250    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e99d3d-66dc-42bd-96db-8b5ae46c592f" containerName="payload-pre-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761255    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0027ba-5a92-4c25-940a-ce550abcfa9f" containerName="payload-processing"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761260    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b560ba-e043-4424-9fc4-962f42f0d321" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761266    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="183f88c0-d9d3-4195-a7a2-a0ed402f2265" containerName="main"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761270    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d3e086-bc80-43e2-aba5-190c8ff3cc4a" containerName="manager"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761276    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1735e4-781b-437e-8b5b-3b1ecb784e13" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761285    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761293    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b59509-9b9a-4399-9f13-4453158404e4" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761299    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761303    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe822e50-9c5e-4233-893a-20179742182d" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761308    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec3b842-e8c3-42fe-a24f-0c3def05b372" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761313    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761317    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba81452-32ec-4672-9892-2dbbabe73285" containerName="maas-api"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761320    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c696c5-8483-48ba-8f51-afafcfb43d9f" containerName="istio-proxy"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761325    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="8002e35d-46ef-4c8f-a3fa-1adbaee4bf18" containerName="main"
Oct 30 15:42:46.761671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.761330    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c861cffd-0939-49dd-8ea8-9f703bb9c942" containerName="istio-proxy"
Oct 30 15:42:46.766524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.766482    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:46.786602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.786571    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:42:46.797907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.797820    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 15:42:46.797907 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.797891    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rwtwc\""
Oct 30 15:42:46.799630 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.799611    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 15:42:46.827217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.827186    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 15:42:46.918432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.918398    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdxf\" (UniqueName: \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:46.918595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.918457    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:46.918595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:46.918524    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.019762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.019667    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdxf\" (UniqueName: \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.019762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.019727    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.019762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.019749    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.022392 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.022360    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.022520 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.022364    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.044448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.044414    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdxf\" (UniqueName: \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\") pod \"maas-controller-68588d554c-wnqtj\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") " pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.080725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.080690    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:47.398506 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.398313    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:42:47.400936 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:42:47.400906    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78642844_f7e5_4cef_93b1_7080656f1c92.slice/crio-e5cd92d900946e070239fcf8ba3fd61b72a3c9283e36ccace596dc8c805b77e2 WatchSource:0}: Error finding container e5cd92d900946e070239fcf8ba3fd61b72a3c9283e36ccace596dc8c805b77e2: Status 404 returned error can't find the container with id e5cd92d900946e070239fcf8ba3fd61b72a3c9283e36ccace596dc8c805b77e2
Oct 30 15:42:47.958710 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.958672    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-wnqtj" event={"ID":"78642844-f7e5-4cef-93b1-7080656f1c92","Type":"ContainerStarted","Data":"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"}
Oct 30 15:42:47.959161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.958732    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-wnqtj" event={"ID":"78642844-f7e5-4cef-93b1-7080656f1c92","Type":"ContainerStarted","Data":"e5cd92d900946e070239fcf8ba3fd61b72a3c9283e36ccace596dc8c805b77e2"}
Oct 30 15:42:47.959161 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:47.958758    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:48.012332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:48.012270    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68588d554c-wnqtj" podStartSLOduration=1.5850818740000001 podStartE2EDuration="2.012250782s" totalImagesPullingTime="427.168908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:48.012155586 +0000 UTC m=+3317.454046304" watchObservedRunningTime="2026-10-30 15:42:48.012250782 +0000 UTC m=+3317.454141499"
Oct 30 15:42:58.966752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:42:58.966713    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:42:59.631264 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:42:59.631223    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:43:13.564636 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.564597    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"]
Oct 30 15:43:13.565193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.565104    5064 cpu_manager.go:414] "containerMap: removing container" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:43:13.565193 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.565120    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d275e859-a3c6-4d17-97b9-3bd90f0aa285" containerName="istio-proxy"
Oct 30 15:43:13.565279 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.565202    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="764bb822-4d34-40e2-aab0-b0b5933faf56" containerName="istio-proxy"
Oct 30 15:43:13.565279 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.565214    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="227280a0-0c75-47d4-81c2-8c2807a4ea26" containerName="istio-proxy"
Oct 30 15:43:13.568680 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.568657    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.609317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.609281    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"]
Oct 30 15:43:13.684789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.684747    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.684803    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.684911    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.684959    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ttn\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-kube-api-access-t4ttn\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.685020    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.685084    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.685134    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.685165    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.685323 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.685236    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.785792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785753    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.785792 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785793    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ttn\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-kube-api-access-t4ttn\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785824    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785888    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785906    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.785932    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786034    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786096    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786139    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786351    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786448 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786432    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786525    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786570 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786549    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.786975 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.786952    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.788288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.788265    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.788548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.788528    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.823018 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.822942    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.823172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.823033    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ttn\" (UniqueName: \"kubernetes.io/projected/59fca057-7aa9-41ad-aaec-f00a0a6754a0-kube-api-access-t4ttn\") pod \"maas-default-gateway-openshift-default-6cc458d887-pg2k9\" (UID: \"59fca057-7aa9-41ad-aaec-f00a0a6754a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:13.882548 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:13.882507    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:14.200797 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:14.200768    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"]
Oct 30 15:43:14.202976 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:43:14.202944    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fca057_7aa9_41ad_aaec_f00a0a6754a0.slice/crio-906de8680a1662fc917c761b8b25d524a1e2bc4292f459f7d71962506201c832 WatchSource:0}: Error finding container 906de8680a1662fc917c761b8b25d524a1e2bc4292f459f7d71962506201c832: Status 404 returned error can't find the container with id 906de8680a1662fc917c761b8b25d524a1e2bc4292f459f7d71962506201c832
Oct 30 15:43:14.205086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:14.205054    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:43:14.205155 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:14.205122    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:43:14.205199 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:14.205156    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:43:15.095046 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.095006    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" event={"ID":"59fca057-7aa9-41ad-aaec-f00a0a6754a0","Type":"ContainerStarted","Data":"bd725a04abcc91f5f65cb8fe9a7de3ae8b2a621d2e3da570cb3ae6e686794767"}
Oct 30 15:43:15.095498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.095071    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" event={"ID":"59fca057-7aa9-41ad-aaec-f00a0a6754a0","Type":"ContainerStarted","Data":"906de8680a1662fc917c761b8b25d524a1e2bc4292f459f7d71962506201c832"}
Oct 30 15:43:15.167259 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.167200    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podStartSLOduration=2.167185099 podStartE2EDuration="2.167185099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:15.166188713 +0000 UTC m=+3344.608079432" watchObservedRunningTime="2026-10-30 15:43:15.167185099 +0000 UTC m=+3344.609075818"
Oct 30 15:43:15.883529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.883492    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:15.885253 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.885226    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:15.885384 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:15.885276    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:16.883552 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:16.883515    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:16.883978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:16.883572    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:17.883389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:17.883350    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:17.883677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:17.883414    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:18.883166 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:18.883121    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:18.883399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:18.883184    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:19.883234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:19.883197    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:19.883677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:19.883256    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:20.883104 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:20.883067    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 15:43:20.883293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:20.883127    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 15:43:21.047082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.047046    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:43:21.047283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.047248    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68588d554c-wnqtj" podUID="78642844-f7e5-4cef-93b1-7080656f1c92" containerName="manager" containerID="cri-o://3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0" gracePeriod=30
Oct 30 15:43:21.307142 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.307112    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:43:21.481642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.481533    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\") pod \"78642844-f7e5-4cef-93b1-7080656f1c92\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") "
Oct 30 15:43:21.481642 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.481612    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\" (UniqueName: \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\") pod \"78642844-f7e5-4cef-93b1-7080656f1c92\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") "
Oct 30 15:43:21.481950 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.481661    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\") pod \"78642844-f7e5-4cef-93b1-7080656f1c92\" (UID: \"78642844-f7e5-4cef-93b1-7080656f1c92\") "
Oct 30 15:43:21.483945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.483910    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert" pod "78642844-f7e5-4cef-93b1-7080656f1c92" (UID: "78642844-f7e5-4cef-93b1-7080656f1c92"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:43:21.483945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.483930    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs" pod "78642844-f7e5-4cef-93b1-7080656f1c92" (UID: "78642844-f7e5-4cef-93b1-7080656f1c92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:43:21.484101 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.483971    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf" pod "78642844-f7e5-4cef-93b1-7080656f1c92" (UID: "78642844-f7e5-4cef-93b1-7080656f1c92"). InnerVolumeSpecName "kube-api-access-4tdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:43:21.582727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.582684    5064 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-webhook-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:43:21.582727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.582717    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tdxf\" (UniqueName: \"kubernetes.io/projected/78642844-f7e5-4cef-93b1-7080656f1c92-kube-api-access-4tdxf\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:43:21.582727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:21.582730    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78642844-f7e5-4cef-93b1-7080656f1c92-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:43:22.149926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.149883    5064 generic.go:356] "Generic (PLEG): container finished" podID="78642844-f7e5-4cef-93b1-7080656f1c92" containerID="3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0" exitCode=0
Oct 30 15:43:22.150370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.149947    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-wnqtj"
Oct 30 15:43:22.150370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.149958    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-wnqtj" event={"ID":"78642844-f7e5-4cef-93b1-7080656f1c92","Type":"ContainerDied","Data":"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"}
Oct 30 15:43:22.150370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.150012    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-wnqtj" event={"ID":"78642844-f7e5-4cef-93b1-7080656f1c92","Type":"ContainerDied","Data":"e5cd92d900946e070239fcf8ba3fd61b72a3c9283e36ccace596dc8c805b77e2"}
Oct 30 15:43:22.150370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.150043    5064 scope.go:122] "RemoveContainer" containerID="3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"
Oct 30 15:43:22.162354 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.162328    5064 scope.go:122] "RemoveContainer" containerID="3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"
Oct 30 15:43:22.162735 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:43:22.162714    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0\": container with ID starting with 3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0 not found: ID does not exist" containerID="3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"
Oct 30 15:43:22.162810 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.162749    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0"} err="failed to get container status \"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0\": rpc error: code = NotFound desc = could not find container \"3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0\": container with ID starting with 3331fe19ceb2759d15bdf4eb9598f97afba88c63554be169b3943377492009c0 not found: ID does not exist"
Oct 30 15:43:22.207733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.207693    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:43:22.222487 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.222445    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68588d554c-wnqtj"]
Oct 30 15:43:22.573749 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.573654    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:22.574111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.574087    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:22.574978 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.574959    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9"
Oct 30 15:43:22.722029 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:22.721994    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78642844-f7e5-4cef-93b1-7080656f1c92" path="/var/lib/kubelet/pods/78642844-f7e5-4cef-93b1-7080656f1c92/volumes"
Oct 30 15:43:23.520911 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.520873    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68588d554c-9tr9l"]
Oct 30 15:43:23.521349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.521334    5064 cpu_manager.go:414] "containerMap: removing container" podUID="78642844-f7e5-4cef-93b1-7080656f1c92" containerName="manager"
Oct 30 15:43:23.521349 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.521350    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78642844-f7e5-4cef-93b1-7080656f1c92" containerName="manager"
Oct 30 15:43:23.521441 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.521434    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="78642844-f7e5-4cef-93b1-7080656f1c92" containerName="manager"
Oct 30 15:43:23.525963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.525939    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.545118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.545083    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rwtwc\""
Oct 30 15:43:23.547437 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.547405    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-9tr9l"]
Oct 30 15:43:23.558576 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.558542    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 15:43:23.645274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.645236    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 15:43:23.645274 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.645269    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 15:43:23.708031 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.707990    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-metrics-certs\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.708313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.708058    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lr9\" (UniqueName: \"kubernetes.io/projected/88625478-51b5-4938-a65c-8665f086e8b2-kube-api-access-s6lr9\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.708313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.708181    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-webhook-cert\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.809294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.809202    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-metrics-certs\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.809294 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.809270    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lr9\" (UniqueName: \"kubernetes.io/projected/88625478-51b5-4938-a65c-8665f086e8b2-kube-api-access-s6lr9\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.809489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.809303    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-webhook-cert\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.811795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.811763    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-webhook-cert\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.811795 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.811784    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88625478-51b5-4938-a65c-8665f086e8b2-metrics-certs\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.834668 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.834637    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lr9\" (UniqueName: \"kubernetes.io/projected/88625478-51b5-4938-a65c-8665f086e8b2-kube-api-access-s6lr9\") pod \"maas-controller-68588d554c-9tr9l\" (UID: \"88625478-51b5-4938-a65c-8665f086e8b2\") " pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:23.846890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:23.846858    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:24.157983 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:24.157954    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68588d554c-9tr9l"]
Oct 30 15:43:24.160105 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:43:24.160077    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88625478_51b5_4938_a65c_8665f086e8b2.slice/crio-828117977c01f7c5a9096615b646cdc99e5538f42f09fbb01fe858c0680abc8d WatchSource:0}: Error finding container 828117977c01f7c5a9096615b646cdc99e5538f42f09fbb01fe858c0680abc8d: Status 404 returned error can't find the container with id 828117977c01f7c5a9096615b646cdc99e5538f42f09fbb01fe858c0680abc8d
Oct 30 15:43:25.168365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:25.168313    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9tr9l" event={"ID":"88625478-51b5-4938-a65c-8665f086e8b2","Type":"ContainerStarted","Data":"07a9a40259ffd13c62d26a7ce2786969f3c2684b90ef6260191d0649f18ad98e"}
Oct 30 15:43:25.168949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:25.168379    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68588d554c-9tr9l" event={"ID":"88625478-51b5-4938-a65c-8665f086e8b2","Type":"ContainerStarted","Data":"828117977c01f7c5a9096615b646cdc99e5538f42f09fbb01fe858c0680abc8d"}
Oct 30 15:43:25.168949 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:25.168502    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:25.216633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:25.216575    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68588d554c-9tr9l" podStartSLOduration=1.792383376 podStartE2EDuration="2.21656082s" totalImagesPullingTime="424.177444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:25.216295377 +0000 UTC m=+3354.658186094" watchObservedRunningTime="2026-10-30 15:43:25.21656082 +0000 UTC m=+3354.658451535"
Oct 30 15:43:29.629904 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:43:29.629868    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:43:36.177097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:43:36.177059    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68588d554c-9tr9l"
Oct 30 15:43:59.629109 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:43:59.629022    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:44:13.793285 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.793248    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"]
Oct 30 15:44:13.797535 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.797515    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.817819 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.817793    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.817970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.817870    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.817970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.817921    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.817970 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.817958    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.818129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.817988    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqlj\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-kube-api-access-pkqlj\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.818129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.818008    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.818129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.818037    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.818129 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.818118    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.818287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.818188    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.826621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.826595    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"]
Oct 30 15:44:13.919820 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919782    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919867    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919901    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919924    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqlj\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-kube-api-access-pkqlj\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919950    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.919985    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920014    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920055    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920224    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920460    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920600 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920514    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920622    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.920747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.920721    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.921032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.921010    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.922368 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.922348    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.922789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.922765    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.947331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.947295    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:13.947543 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:13.947517    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqlj\" (UniqueName: \"kubernetes.io/projected/d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a-kube-api-access-pkqlj\") pod \"maas-default-gateway-openshift-default-6cc458d887-bfgkh\" (UID: \"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:14.113335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:14.113224    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:14.445687 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:14.445317    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"]
Oct 30 15:44:14.448191 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:44:14.448145    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e5ffd6_da5b_4fc6_a400_cfa4acf4887a.slice/crio-fa3f5201d76393c80481e055706ee98d6f2e5abe571e29cfe2cb333afd96e1fb WatchSource:0}: Error finding container fa3f5201d76393c80481e055706ee98d6f2e5abe571e29cfe2cb333afd96e1fb: Status 404 returned error can't find the container with id fa3f5201d76393c80481e055706ee98d6f2e5abe571e29cfe2cb333afd96e1fb
Oct 30 15:44:14.450727 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:14.450688    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:14.450866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:14.450776    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:14.450969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:14.450822    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:15.440072 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:15.440025    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" event={"ID":"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a","Type":"ContainerStarted","Data":"3cf0ea1eadf7815d65b5645d15c5435c460bab399bd4d0f09b89b30bf991c0b8"}
Oct 30 15:44:15.440596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:15.440100    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" event={"ID":"d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a","Type":"ContainerStarted","Data":"fa3f5201d76393c80481e055706ee98d6f2e5abe571e29cfe2cb333afd96e1fb"}
Oct 30 15:44:15.532864 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:15.532765    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podStartSLOduration=2.532742955 podStartE2EDuration="2.532742955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:15.501038067 +0000 UTC m=+3404.942928783" watchObservedRunningTime="2026-10-30 15:44:15.532742955 +0000 UTC m=+3404.974633675"
Oct 30 15:44:16.132854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:16.123644    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:16.132854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:16.123780    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:16.132854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:16.123852    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:17.117263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:17.117220    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:17.117926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:17.117890    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:18.117396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:18.117354    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:18.118081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:18.118034    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:19.117854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:19.117301    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:19.117854 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:19.117365    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:20.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:20.118278    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:20.127855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:20.118357    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:21.114085 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:21.114043    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 15:44:21.114278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:21.114114    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 15:44:23.113954 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:23.113905    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:23.114404 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:23.113983    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:24.114568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:24.114519    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:24.115143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:24.114595    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:24.885443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:24.885406    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:24.885890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:24.885859    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:24.886623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:24.886604    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh"
Oct 30 15:44:29.631308 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:44:29.631265    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:44:33.872155 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:33.872112    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 15:44:33.872771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:33.872330    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerName="discovery" containerID="cri-o://fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f" gracePeriod=30
Oct 30 15:44:34.139603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.139529    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 15:44:34.290880 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.290819    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.290900    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.290963    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.290981    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.291034    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.291055    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.291077    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\") pod \"0e567580-1d49-4824-ac97-23eff3ab24b0\" (UID: \"0e567580-1d49-4824-ac97-23eff3ab24b0\") "
Oct 30 15:44:34.291442 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.291339    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:44:34.291499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.291477    5064 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-ca-configmap\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.293527 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.293501    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:44:34.293762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.293730    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "kube-api-access-wcs47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:44:34.293923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.293766    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:44:34.293923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.293815    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:44:34.293923 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.293811    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:44:34.297347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.297304    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts" pod "0e567580-1d49-4824-ac97-23eff3ab24b0" (UID: "0e567580-1d49-4824-ac97-23eff3ab24b0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393056    5064 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-kubeconfig\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393100    5064 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-csr-dns-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393115    5064 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e567580-1d49-4824-ac97-23eff3ab24b0-local-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393131    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcs47\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-kube-api-access-wcs47\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393143    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e567580-1d49-4824-ac97-23eff3ab24b0-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.393153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.393151    5064 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e567580-1d49-4824-ac97-23eff3ab24b0-cacerts\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:44:34.542651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.542617    5064 generic.go:356] "Generic (PLEG): container finished" podID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerID="fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f" exitCode=0
Oct 30 15:44:34.542867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.542667    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"
Oct 30 15:44:34.542867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.542709    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" event={"ID":"0e567580-1d49-4824-ac97-23eff3ab24b0","Type":"ContainerDied","Data":"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"}
Oct 30 15:44:34.542867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.542765    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8" event={"ID":"0e567580-1d49-4824-ac97-23eff3ab24b0","Type":"ContainerDied","Data":"5fcc3dfa72d834652e01816774a96eab4bc603de9bcc3df3fb65d95f0ed8ec77"}
Oct 30 15:44:34.542867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.542786    5064 scope.go:122] "RemoveContainer" containerID="fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"
Oct 30 15:44:34.561583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.561397    5064 scope.go:122] "RemoveContainer" containerID="fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"
Oct 30 15:44:34.561778 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:44:34.561753    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f\": container with ID starting with fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f not found: ID does not exist" containerID="fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"
Oct 30 15:44:34.561894 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.561786    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f"} err="failed to get container status \"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f\": rpc error: code = NotFound desc = could not find container \"fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f\": container with ID starting with fe4e68301f8a311be305b4ba544cca8088672cc01d0afeb599d449c3de162b8f not found: ID does not exist"
Oct 30 15:44:34.604886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.604855    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 15:44:34.618620 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.618592    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vbpm8"]
Oct 30 15:44:34.722373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:34.722290    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" path="/var/lib/kubelet/pods/0e567580-1d49-4824-ac97-23eff3ab24b0/volumes"
Oct 30 15:44:43.852200 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.852161    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"]
Oct 30 15:44:43.852666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.852645    5064 cpu_manager.go:414] "containerMap: removing container" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerName="discovery"
Oct 30 15:44:43.852666 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.852660    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerName="discovery"
Oct 30 15:44:43.852754 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.852735    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e567580-1d49-4824-ac97-23eff3ab24b0" containerName="discovery"
Oct 30 15:44:43.857318 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.857294    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.891612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.891571    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"]
Oct 30 15:44:43.924590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924550    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924591    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924708    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924746    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924883    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924915    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxrh\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-kube-api-access-nzxrh\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.924963 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.924943    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.925065 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.925052    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:43.925103 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:43.925083    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5234d7a7-d39b-431b-a017-aa0498cb5b33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026289 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026243    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026302    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026372    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026401    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxrh\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-kube-api-access-nzxrh\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026430    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026499 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026485    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026526    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5234d7a7-d39b-431b-a017-aa0498cb5b33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026582    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026607    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.026789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026771    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.027067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026819    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.027067 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.026931    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.027191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.027102    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.027413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.027388    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5234d7a7-d39b-431b-a017-aa0498cb5b33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.029007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.028988    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.029263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.029241    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.051737 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.051699    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.051912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.051782    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxrh\" (UniqueName: \"kubernetes.io/projected/5234d7a7-d39b-431b-a017-aa0498cb5b33-kube-api-access-nzxrh\") pod \"maas-default-gateway-openshift-default-6cc458d887-2sncc\" (UID: \"5234d7a7-d39b-431b-a017-aa0498cb5b33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.170399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.170355    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:44.490616 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.490584    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"]
Oct 30 15:44:44.493017 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:44:44.492983    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5234d7a7_d39b_431b_a017_aa0498cb5b33.slice/crio-5e7a1a0517c2f38cc06fab1c5a05f82b96e5d22e92eb421e68dbb9e160b2d287 WatchSource:0}: Error finding container 5e7a1a0517c2f38cc06fab1c5a05f82b96e5d22e92eb421e68dbb9e160b2d287: Status 404 returned error can't find the container with id 5e7a1a0517c2f38cc06fab1c5a05f82b96e5d22e92eb421e68dbb9e160b2d287
Oct 30 15:44:44.495249 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.495216    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:44.495321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.495282    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:44.495321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.495311    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:44:44.622720 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.622687    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" event={"ID":"5234d7a7-d39b-431b-a017-aa0498cb5b33","Type":"ContainerStarted","Data":"0a283898505b70bea70598028b8a7d27c9651e5496ddb8ed3d6ae0c422b56c41"}
Oct 30 15:44:44.622945 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.622740    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" event={"ID":"5234d7a7-d39b-431b-a017-aa0498cb5b33","Type":"ContainerStarted","Data":"5e7a1a0517c2f38cc06fab1c5a05f82b96e5d22e92eb421e68dbb9e160b2d287"}
Oct 30 15:44:44.671066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:44.671001    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podStartSLOduration=1.67098758 podStartE2EDuration="1.67098758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:44.670526858 +0000 UTC m=+3434.112417574" watchObservedRunningTime="2026-10-30 15:44:44.67098758 +0000 UTC m=+3434.112878297"
Oct 30 15:44:45.170887 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:45.170844    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:45.172647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:45.172620    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:45.172786 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:45.172671    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:46.171386 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:46.171344    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:46.171803 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:46.171405    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:47.171179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:47.171139    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:47.171362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:47.171196    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:48.171194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:48.171157    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:48.171656 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:48.171214    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:49.170818 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:49.170774    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:49.171064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:49.170865    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:50.171615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:50.171576    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:50.172043 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:50.171630    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:51.171045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:51.171006    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 15:44:51.171243 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:51.171065    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 15:44:53.172242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:53.172163    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:53.172808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:53.172250    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:54.226003 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:54.225946    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:54.226707 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:54.226667    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:55.272868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.272013    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:55.272868 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.272087    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:55.615034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.614534    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:44:55.615034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.614606    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:44:55.805228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.805192    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:55.805615 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.805592    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:55.806330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:44:55.806312    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc"
Oct 30 15:44:59.631301 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:44:59.631259    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:45:23.621855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:45:23.620939    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:45:23.621855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:45:23.621012    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:45:29.631134 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:45:29.631090    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:45:38.630032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:45:38.629969    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pg2k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:45:38.630865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:45:38.630798    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pg2k9" podUID="59fca057-7aa9-41ad-aaec-f00a0a6754a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:45:59.628940 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:45:59.628890    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:46:29.630758 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:46:29.630714    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:46:40.653090 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:46:40.653009    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9vjxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:46:40.653977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:46:40.653914    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9vjxl" podUID="982473c9-6215-4df5-8a3f-6d88ffbac6e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:46:40.935333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:46:40.935185    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bfgkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:46:40.935806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:46:40.935639    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bfgkh" podUID="d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:46:59.630397 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:46:59.630342    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:47:29.629378 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:47:29.629335    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:47:31.437430 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:47:31.437402    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:47:31.463611 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:47:31.463585    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:47:59.629506 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:47:59.629468    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:48:02.282755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.282717    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:48:02.286938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.286913    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.308305 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.308265    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:48:02.318168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.318139    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c2454e-openshift-default-dockercfg-7jk5t\""
Oct 30 15:48:02.329718 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329689    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.329916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329728    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.329916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329757    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps2x\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.329916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329896    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.330058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329917    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.330058 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.329945    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.330134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.330059    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.330134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.330085    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.330204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.330137    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.430756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.430714    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.430756 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.430760    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.430873    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.430998    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431036    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431153 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431070    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431152    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431186    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431278 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431224    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps2x\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431413    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431505 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431443    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431734 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431710    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431804    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.431896 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.431877    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.433415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.433392    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.433651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.433634    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.464415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.464383    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps2x\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.464562 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.464388    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\") pod \"e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") " pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.607055 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.606943    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:02.925920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.925889    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:48:02.928491 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:48:02.928453    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cc6c2d_d2b4_49fe_a8a2_3d4f315a1f53.slice/crio-53306f4cd6926d1096615a8cbe002ea42833ddd4bcab040ea979fd5c86df0e92 WatchSource:0}: Error finding container 53306f4cd6926d1096615a8cbe002ea42833ddd4bcab040ea979fd5c86df0e92: Status 404 returned error can't find the container with id 53306f4cd6926d1096615a8cbe002ea42833ddd4bcab040ea979fd5c86df0e92
Oct 30 15:48:02.930855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.930796    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:02.930943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.930922    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:02.930988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:02.930963    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:03.873771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:03.873733    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerStarted","Data":"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"}
Oct 30 15:48:03.874263 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:03.873794    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerStarted","Data":"53306f4cd6926d1096615a8cbe002ea42833ddd4bcab040ea979fd5c86df0e92"}
Oct 30 15:48:03.936990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:03.936931    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podStartSLOduration=1.936916161 podStartE2EDuration="1.936916161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:03.935517497 +0000 UTC m=+3633.377408216" watchObservedRunningTime="2026-10-30 15:48:03.936916161 +0000 UTC m=+3633.378806857"
Oct 30 15:48:04.607993 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:04.607952    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:04.613574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:04.613544    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:04.879302 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:04.879266    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:04.880465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:04.880435    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:07.872336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.872290    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:48:07.876330 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.876305    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:07.893377 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.893349    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c2454e-serving-cert\""
Oct 30 15:48:07.911143 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.911115    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:48:07.993389 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.993357    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:07.993581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.993419    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:07.993646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.993621    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:07.993692 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:07.993659    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzd8\" (UniqueName: \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.095167 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.095123    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.095342 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.095204    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.095399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.095359    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.095399 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.095382    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzd8\" (UniqueName: \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.096164 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.096132    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.097762 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.097740    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.097998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.097976    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.123005 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.122926    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzd8\" (UniqueName: \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\") pod \"maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.190203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.190167    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:08.362695 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.362657    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:48:08.366601 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.366584    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.386724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.386698    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-76ec15b8-dockercfg-7hfln\""
Oct 30 15:48:08.386724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.386687    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c2454e\""
Oct 30 15:48:08.388998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.388973    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:48:08.399352 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.399329    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.399489 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.399395    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldljw\" (UniqueName: \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.408926 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.408891    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:48:08.412495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.412475    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.436999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.436972    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:48:08.500846 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.500782    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.501106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.500918    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzbn\" (UniqueName: \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.501106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.500972    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldljw\" (UniqueName: \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.501106 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.501034    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.502733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.502052    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.517544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.517519    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:48:08.520282 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:48:08.520251    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0e258e_1085_413e_afce_21e4e28eca64.slice/crio-45330b9d3e0a3c19e37e370c334e5e8489fa32fc6d2330ba702b1318cba92b0e WatchSource:0}: Error finding container 45330b9d3e0a3c19e37e370c334e5e8489fa32fc6d2330ba702b1318cba92b0e: Status 404 returned error can't find the container with id 45330b9d3e0a3c19e37e370c334e5e8489fa32fc6d2330ba702b1318cba92b0e
Oct 30 15:48:08.521635 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.521615    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:48:08.526488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.526467    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldljw\" (UniqueName: \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\") pod \"payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.601715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.601673    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.601938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.601808    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzbn\" (UniqueName: \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.602428 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.602409    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.626603 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.626534    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzbn\" (UniqueName: \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\") pod \"payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.680524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.680475    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:08.726681 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.726466    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:08.900127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.900096    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" event={"ID":"7f0e258e-1085-413e-afce-21e4e28eca64","Type":"ContainerStarted","Data":"45330b9d3e0a3c19e37e370c334e5e8489fa32fc6d2330ba702b1318cba92b0e"}
Oct 30 15:48:08.983508 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:08.983478    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:48:08.985945 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:48:08.985916    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcb8d4f_6b25_4461_a621_3e39dc831515.slice/crio-98574e32cc2c6828708cec9fe0356d5c84b815ab07a9f9eb6ccaab4677bf1540 WatchSource:0}: Error finding container 98574e32cc2c6828708cec9fe0356d5c84b815ab07a9f9eb6ccaab4677bf1540: Status 404 returned error can't find the container with id 98574e32cc2c6828708cec9fe0356d5c84b815ab07a9f9eb6ccaab4677bf1540
Oct 30 15:48:09.040151 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.040125    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:48:09.041064 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:48:09.041041    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1f336_1bba_4443_8651_7fbf30a4107c.slice/crio-949ad1cd3d9497360547cb23c8183225b812b1f8016eec0fcf525576e047e96e WatchSource:0}: Error finding container 949ad1cd3d9497360547cb23c8183225b812b1f8016eec0fcf525576e047e96e: Status 404 returned error can't find the container with id 949ad1cd3d9497360547cb23c8183225b812b1f8016eec0fcf525576e047e96e
Oct 30 15:48:09.907725 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.907689    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" event={"ID":"cfcb8d4f-6b25-4461-a621-3e39dc831515","Type":"ContainerStarted","Data":"cc124d3560ad89efdfcaf817cf1ef98a208adc4095c646dd96a9bbe0d650f35b"}
Oct 30 15:48:09.908214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.907753    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" event={"ID":"cfcb8d4f-6b25-4461-a621-3e39dc831515","Type":"ContainerStarted","Data":"98574e32cc2c6828708cec9fe0356d5c84b815ab07a9f9eb6ccaab4677bf1540"}
Oct 30 15:48:09.908214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.907856    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:09.909307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.909280    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" event={"ID":"7f0e258e-1085-413e-afce-21e4e28eca64","Type":"ContainerStarted","Data":"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"}
Oct 30 15:48:09.909422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.909336    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:09.910861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.910825    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" event={"ID":"34b1f336-1bba-4443-8651-7fbf30a4107c","Type":"ContainerStarted","Data":"bfef444dbfe4e6423654627db4810cc8acfc3bba6df68778796341cd48190325"}
Oct 30 15:48:09.910991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.910876    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" event={"ID":"34b1f336-1bba-4443-8651-7fbf30a4107c","Type":"ContainerStarted","Data":"949ad1cd3d9497360547cb23c8183225b812b1f8016eec0fcf525576e047e96e"}
Oct 30 15:48:09.910991 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.910904    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:09.952010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.951952    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" podStartSLOduration=1.951937671 podStartE2EDuration="1.951937671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:09.950657547 +0000 UTC m=+3639.392548291" watchObservedRunningTime="2026-10-30 15:48:09.951937671 +0000 UTC m=+3639.393828387"
Oct 30 15:48:09.994741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:09.994684    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" podStartSLOduration=1.994671425 podStartE2EDuration="1.994671425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:09.994449496 +0000 UTC m=+3639.436340213" watchObservedRunningTime="2026-10-30 15:48:09.994671425 +0000 UTC m=+3639.436562141"
Oct 30 15:48:10.045242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:10.045190    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" podStartSLOduration=2.672922699 podStartE2EDuration="3.045174274s" totalImagesPullingTime="372.251575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:10.043122434 +0000 UTC m=+3639.485013152" watchObservedRunningTime="2026-10-30 15:48:10.045174274 +0000 UTC m=+3639.487064987"
Oct 30 15:48:15.940612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:15.940516    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:48:20.919690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:20.919659    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:48:20.920264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:20.919736    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:48:20.921974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:20.921943    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:48:20.922105 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:20.922011    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:48:21.993321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.993233    5064 generic.go:356] "Generic (PLEG): container finished" podID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerID="40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf" exitCode=0
Oct 30 15:48:21.993321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.993295    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerDied","Data":"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"}
Oct 30 15:48:21.993799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.993738    5064 scope.go:122] "RemoveContainer" containerID="40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"
Oct 30 15:48:21.995091 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.995038    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:21.995182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.995135    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:21.995182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:21.995169    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:48:23.000747 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.000705    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerStarted","Data":"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"}
Oct 30 15:48:23.608097 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.608055    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:23.608303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.608278    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 15:48:23.608357 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.608328    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 15:48:23.842303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.842172    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:48:23.846799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.846768    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.864871 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.864748    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c2454e\"/\"openshift-service-ca.crt\""
Oct 30 15:48:23.865036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.864914    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c2454e\"/\"kube-root-ca.crt\""
Oct 30 15:48:23.868602 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.868577    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:48:23.937920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.929204    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c2454e\"/\"simulator-single-main-c2454e-kserve-self-signed-certs\""
Oct 30 15:48:23.937920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.929365    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c2454e\"/\"default-dockercfg-cv8pr\""
Oct 30 15:48:23.976284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976234    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.976284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976287    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.976583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976446    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.976583 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976507    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5xb\" (UniqueName: \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.976701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976604    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:23.976701 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:23.976694    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.077874 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.077815    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.077956    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.077993    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.078020    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.078106    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078415 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.078151    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5xb\" (UniqueName: \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.078767 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.078746    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.080409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.079788    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.080409 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.080367    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.089883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.088186    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.091380 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.091350    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.105558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.105485    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5xb\" (UniqueName: \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\") pod \"simulator-single-main-c2454e-kserve-c7bc777c9-bgph8\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") " pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.170497 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.170452    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:24.496137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.496103    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:48:24.498597 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:48:24.498564    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b268107_de4f_4217_8463_2d075821e072.slice/crio-768ca9789dfdb5291236725d4d27145795466426ce05e99182c0eca51e2e0066 WatchSource:0}: Error finding container 768ca9789dfdb5291236725d4d27145795466426ce05e99182c0eca51e2e0066: Status 404 returned error can't find the container with id 768ca9789dfdb5291236725d4d27145795466426ce05e99182c0eca51e2e0066
Oct 30 15:48:24.607424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.607390    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 15:48:24.607574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:24.607448    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 15:48:25.012317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.012284    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" event={"ID":"6b268107-de4f-4217-8463-2d075821e072","Type":"ContainerStarted","Data":"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"}
Oct 30 15:48:25.012525 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.012327    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" event={"ID":"6b268107-de4f-4217-8463-2d075821e072","Type":"ContainerStarted","Data":"768ca9789dfdb5291236725d4d27145795466426ce05e99182c0eca51e2e0066"}
Oct 30 15:48:25.067596 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.067539    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" podStartSLOduration=2.06752299 podStartE2EDuration="2.06752299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:25.065863525 +0000 UTC m=+3654.507754237" watchObservedRunningTime="2026-10-30 15:48:25.06752299 +0000 UTC m=+3654.509413707"
Oct 30 15:48:25.171563 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.171522    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:25.608203 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.608156    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 15:48:25.608400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:25.608225    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 15:48:26.608157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:26.608119    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 15:48:26.608654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:26.608183    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 15:48:27.607733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:27.607689    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 15:48:27.607974 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:27.607752    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 15:48:28.632739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:28.632707    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:28.633486 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:28.633435    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:28.634775 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:28.634753    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:48:29.631395 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:48:29.631357    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:48:34.171011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:34.170968    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:34.183575 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:34.183545    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:35.105553 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:48:35.105513    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:48:59.630977 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:48:59.630929    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:49:03.528411 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:03.528369    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:49:09.117037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:09.116997    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:49:09.150445 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:09.150404    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:49:09.713080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:09.713043    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:49:09.713284 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:09.713253    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" podUID="7f0e258e-1085-413e-afce-21e4e28eca64" containerName="maas-api" containerID="cri-o://3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f" gracePeriod=30
Oct 30 15:49:10.216351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.216320    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:49:10.288529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.288436    5064 generic.go:356] "Generic (PLEG): container finished" podID="7f0e258e-1085-413e-afce-21e4e28eca64" containerID="3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f" exitCode=0
Oct 30 15:49:10.288529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.288489    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"
Oct 30 15:49:10.288772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.288526    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" event={"ID":"7f0e258e-1085-413e-afce-21e4e28eca64","Type":"ContainerDied","Data":"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"}
Oct 30 15:49:10.288772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.288615    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2" event={"ID":"7f0e258e-1085-413e-afce-21e4e28eca64","Type":"ContainerDied","Data":"45330b9d3e0a3c19e37e370c334e5e8489fa32fc6d2330ba702b1318cba92b0e"}
Oct 30 15:49:10.288772 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.288647    5064 scope.go:122] "RemoveContainer" containerID="3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"
Oct 30 15:49:10.300717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.300693    5064 scope.go:122] "RemoveContainer" containerID="3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"
Oct 30 15:49:10.301153 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:10.301120    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f\": container with ID starting with 3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f not found: ID does not exist" containerID="3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"
Oct 30 15:49:10.301250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.301165    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f"} err="failed to get container status \"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f\": rpc error: code = NotFound desc = could not find container \"3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f\": container with ID starting with 3770fd6c0ea37f6895afaa4b8d3bb8175260ecdbd44a66087df3dfed803d274f not found: ID does not exist"
Oct 30 15:49:10.314111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.314081    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\") pod \"7f0e258e-1085-413e-afce-21e4e28eca64\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") "
Oct 30 15:49:10.314303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.314140    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\" (UniqueName: \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\") pod \"7f0e258e-1085-413e-afce-21e4e28eca64\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") "
Oct 30 15:49:10.314303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.314182    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\") pod \"7f0e258e-1085-413e-afce-21e4e28eca64\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") "
Oct 30 15:49:10.314303 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.314229    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\") pod \"7f0e258e-1085-413e-afce-21e4e28eca64\" (UID: \"7f0e258e-1085-413e-afce-21e4e28eca64\") "
Oct 30 15:49:10.314669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.314640    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca" pod "7f0e258e-1085-413e-afce-21e4e28eca64" (UID: "7f0e258e-1085-413e-afce-21e4e28eca64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:49:10.316421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.316397    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs" pod "7f0e258e-1085-413e-afce-21e4e28eca64" (UID: "7f0e258e-1085-413e-afce-21e4e28eca64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:49:10.316526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.316423    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls" pod "7f0e258e-1085-413e-afce-21e4e28eca64" (UID: "7f0e258e-1085-413e-afce-21e4e28eca64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:49:10.316526 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.316510    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8" pod "7f0e258e-1085-413e-afce-21e4e28eca64" (UID: "7f0e258e-1085-413e-afce-21e4e28eca64"). InnerVolumeSpecName "kube-api-access-clzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:10.415742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.415704    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:10.415742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.415740    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzd8\" (UniqueName: \"kubernetes.io/projected/7f0e258e-1085-413e-afce-21e4e28eca64-kube-api-access-clzd8\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:10.415742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.415752    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0e258e-1085-413e-afce-21e4e28eca64-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:10.416032 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.415766    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0e258e-1085-413e-afce-21e4e28eca64-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:10.637324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.637286    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:49:10.650989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.650948    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c2454e-bd9f4cc56-24mt2"]
Oct 30 15:49:10.721846 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:10.721809    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0e258e-1085-413e-afce-21e4e28eca64" path="/var/lib/kubelet/pods/7f0e258e-1085-413e-afce-21e4e28eca64/volumes"
Oct 30 15:49:14.117289 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:14.117243    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing" containerID="cri-o://bfef444dbfe4e6423654627db4810cc8acfc3bba6df68778796341cd48190325" gracePeriod=25
Oct 30 15:49:14.151511 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:14.151451    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing" containerID="cri-o://cc124d3560ad89efdfcaf817cf1ef98a208adc4095c646dd96a9bbe0d650f35b" gracePeriod=25
Oct 30 15:49:18.549893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.549815    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" podUID="6b268107-de4f-4217-8463-2d075821e072" containerName="main" containerID="cri-o://9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a" gracePeriod=45
Oct 30 15:49:18.828070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.828038    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:49:18.904189 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904153    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904204    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904286    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904394 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904373    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904451    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904467    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:18.904595 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904494    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\" (UniqueName: \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\") pod \"6b268107-de4f-4217-8463-2d075821e072\" (UID: \"6b268107-de4f-4217-8463-2d075821e072\") "
Oct 30 15:49:18.904771 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904702    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:18.904997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.904974    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:18.905110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.905001    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:18.905110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.905009    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:18.907272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.907239    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:49:18.907391 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.907376    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:18.909982 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:18.909953    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb" pod "6b268107-de4f-4217-8463-2d075821e072" (UID: "6b268107-de4f-4217-8463-2d075821e072"). InnerVolumeSpecName "kube-api-access-sw5xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:19.006293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.006254    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5xb\" (UniqueName: \"kubernetes.io/projected/6b268107-de4f-4217-8463-2d075821e072-kube-api-access-sw5xb\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:19.006293 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.006292    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b268107-de4f-4217-8463-2d075821e072-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:19.006507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.006306    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:19.006507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.006318    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b268107-de4f-4217-8463-2d075821e072-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:19.338006 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.337971    5064 generic.go:356] "Generic (PLEG): container finished" podID="6b268107-de4f-4217-8463-2d075821e072" containerID="9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a" exitCode=0
Oct 30 15:49:19.338210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.338019    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"
Oct 30 15:49:19.338210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.338069    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" event={"ID":"6b268107-de4f-4217-8463-2d075821e072","Type":"ContainerDied","Data":"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"}
Oct 30 15:49:19.338210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.338110    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8" event={"ID":"6b268107-de4f-4217-8463-2d075821e072","Type":"ContainerDied","Data":"768ca9789dfdb5291236725d4d27145795466426ce05e99182c0eca51e2e0066"}
Oct 30 15:49:19.338210 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.338130    5064 scope.go:122] "RemoveContainer" containerID="9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"
Oct 30 15:49:19.348008 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.347992    5064 scope.go:122] "RemoveContainer" containerID="9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"
Oct 30 15:49:19.348296 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:19.348275    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a\": container with ID starting with 9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a not found: ID does not exist" containerID="9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"
Oct 30 15:49:19.348345 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.348304    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a"} err="failed to get container status \"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a\": rpc error: code = NotFound desc = could not find container \"9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a\": container with ID starting with 9a652ef357dea0d4191c32985c1bec0ee114d21f0e06a59dc402dc6a9988ac6a not found: ID does not exist"
Oct 30 15:49:19.414230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.414184    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:49:19.417179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.417138    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c2454e/simulator-single-main-c2454e-kserve-c7bc777c9-bgph8"]
Oct 30 15:49:19.427052 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.427019    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:49:19.427373 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:19.427324    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy" containerID="cri-o://2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77" gracePeriod=30
Oct 30 15:49:20.734124 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:20.734088    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b268107-de4f-4217-8463-2d075821e072" path="/var/lib/kubelet/pods/6b268107-de4f-4217-8463-2d075821e072/volumes"
Oct 30 15:49:20.917872 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:20.917801    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:49:20.918073 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:20.917894    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:49:24.694395 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.694336    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:49:24.771045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771010    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771060    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771082    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771228 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771203    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771367    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771381    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:24.771542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771424    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771542 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771453    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771565    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771602    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\") pod \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\" (UID: \"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53\") "
Oct 30 15:49:24.771789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771451    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:24.771789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.771665    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:24.772037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.772014    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.772037 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.772027    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:49:24.772178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.772043    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.772178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.772044    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:24.772178 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.772058    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.773581 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.773538    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:49:24.773722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.773676    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:24.773912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.773889    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "kube-api-access-tps2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:24.774027 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.774003    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy" pod "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" (UID: "e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:49:24.873393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873364    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tps2x\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-kube-api-access-tps2x\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.873393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873390    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.873393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873400    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.873393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873409    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.873672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873418    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:24.873672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:24.873427    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:25.371623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.371579    5064 generic.go:356] "Generic (PLEG): container finished" podID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerID="2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77" exitCode=0
Oct 30 15:49:25.371623 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.371625    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"
Oct 30 15:49:25.371969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.371671    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerDied","Data":"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"}
Oct 30 15:49:25.371969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.371739    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp" event={"ID":"e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53","Type":"ContainerDied","Data":"53306f4cd6926d1096615a8cbe002ea42833ddd4bcab040ea979fd5c86df0e92"}
Oct 30 15:49:25.371969 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.371769    5064 scope.go:122] "RemoveContainer" containerID="2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"
Oct 30 15:49:25.382028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.382009    5064 scope.go:122] "RemoveContainer" containerID="40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"
Oct 30 15:49:25.391028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.391008    5064 scope.go:122] "RemoveContainer" containerID="2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"
Oct 30 15:49:25.394427 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:25.391714    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77\": container with ID starting with 2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77 not found: ID does not exist" containerID="2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"
Oct 30 15:49:25.394427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.391753    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77"} err="failed to get container status \"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77\": rpc error: code = NotFound desc = could not find container \"2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77\": container with ID starting with 2d52c6b1cf240a30e71a435eee3893fb3e290ad260962af7d29642b024f40e77 not found: ID does not exist"
Oct 30 15:49:25.394427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.391779    5064 scope.go:122] "RemoveContainer" containerID="40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"
Oct 30 15:49:25.394427 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:25.392160    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf\": container with ID starting with 40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf not found: ID does not exist" containerID="40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"
Oct 30 15:49:25.394427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.392190    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf"} err="failed to get container status \"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf\": rpc error: code = NotFound desc = could not find container \"40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf\": container with ID starting with 40890314a7ea7291398a0b69aca3e25b709c1423cd51d9f73427af19b9f4cbaf not found: ID does not exist"
Oct 30 15:49:25.428084 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.428047    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:49:25.441578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:25.441533    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c2454e-openshift-default-7cdf7db7cc7gctp"]
Oct 30 15:49:26.722507 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:26.722474    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" path="/var/lib/kubelet/pods/e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53/volumes"
Oct 30 15:49:29.631107 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:29.631067    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:49:30.918425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:30.918386    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:49:30.918865 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:30.918387    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:49:34.454713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:34.454439    5064 generic.go:356] "Generic (PLEG): container finished" podID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerID="cc124d3560ad89efdfcaf817cf1ef98a208adc4095c646dd96a9bbe0d650f35b" exitCode=0
Oct 30 15:49:34.454713 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:34.454567    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" event={"ID":"cfcb8d4f-6b25-4461-a621-3e39dc831515","Type":"ContainerDied","Data":"cc124d3560ad89efdfcaf817cf1ef98a208adc4095c646dd96a9bbe0d650f35b"}
Oct 30 15:49:34.470454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:34.470168    5064 generic.go:356] "Generic (PLEG): container finished" podID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerID="bfef444dbfe4e6423654627db4810cc8acfc3bba6df68778796341cd48190325" exitCode=0
Oct 30 15:49:34.470454 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:34.470314    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" event={"ID":"34b1f336-1bba-4443-8651-7fbf30a4107c","Type":"ContainerDied","Data":"bfef444dbfe4e6423654627db4810cc8acfc3bba6df68778796341cd48190325"}
Oct 30 15:49:35.448597 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.445152    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:49:35.480658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.480262    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg" event={"ID":"34b1f336-1bba-4443-8651-7fbf30a4107c","Type":"ContainerDied","Data":"949ad1cd3d9497360547cb23c8183225b812b1f8016eec0fcf525576e047e96e"}
Oct 30 15:49:35.480658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.480277    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"
Oct 30 15:49:35.480658 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.480354    5064 scope.go:122] "RemoveContainer" containerID="bfef444dbfe4e6423654627db4810cc8acfc3bba6df68778796341cd48190325"
Oct 30 15:49:35.529992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.529959    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:49:35.537931 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.537894    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\") pod \"34b1f336-1bba-4443-8651-7fbf30a4107c\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") "
Oct 30 15:49:35.538122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.538045    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\" (UniqueName: \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\") pod \"34b1f336-1bba-4443-8651-7fbf30a4107c\" (UID: \"34b1f336-1bba-4443-8651-7fbf30a4107c\") "
Oct 30 15:49:35.538256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.538233    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume" pod "34b1f336-1bba-4443-8651-7fbf30a4107c" (UID: "34b1f336-1bba-4443-8651-7fbf30a4107c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:49:35.538420 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.538403    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1f336-1bba-4443-8651-7fbf30a4107c-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:35.540261 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.540228    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn" pod "34b1f336-1bba-4443-8651-7fbf30a4107c" (UID: "34b1f336-1bba-4443-8651-7fbf30a4107c"). InnerVolumeSpecName "kube-api-access-ktzbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:35.639181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.639143    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\") pod \"cfcb8d4f-6b25-4461-a621-3e39dc831515\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") "
Oct 30 15:49:35.639365 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.639326    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\" (UniqueName: \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\") pod \"cfcb8d4f-6b25-4461-a621-3e39dc831515\" (UID: \"cfcb8d4f-6b25-4461-a621-3e39dc831515\") "
Oct 30 15:49:35.639569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.639544    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume" pod "cfcb8d4f-6b25-4461-a621-3e39dc831515" (UID: "cfcb8d4f-6b25-4461-a621-3e39dc831515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:49:35.639651 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.639636    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzbn\" (UniqueName: \"kubernetes.io/projected/34b1f336-1bba-4443-8651-7fbf30a4107c-kube-api-access-ktzbn\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:35.639699 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.639655    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb8d4f-6b25-4461-a621-3e39dc831515-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:35.641469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.641448    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw" pod "cfcb8d4f-6b25-4461-a621-3e39dc831515" (UID: "cfcb8d4f-6b25-4461-a621-3e39dc831515"). InnerVolumeSpecName "kube-api-access-ldljw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:49:35.740586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.740532    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldljw\" (UniqueName: \"kubernetes.io/projected/cfcb8d4f-6b25-4461-a621-3e39dc831515-kube-api-access-ldljw\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:49:35.830783 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.830737    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:49:35.844378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:35.844342    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c2454e-5968748b68rkvg"]
Oct 30 15:49:36.486456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.486418    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"
Oct 30 15:49:36.486456 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.486441    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb" event={"ID":"cfcb8d4f-6b25-4461-a621-3e39dc831515","Type":"ContainerDied","Data":"98574e32cc2c6828708cec9fe0356d5c84b815ab07a9f9eb6ccaab4677bf1540"}
Oct 30 15:49:36.487036 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.486516    5064 scope.go:122] "RemoveContainer" containerID="cc124d3560ad89efdfcaf817cf1ef98a208adc4095c646dd96a9bbe0d650f35b"
Oct 30 15:49:36.542234 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.542199    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:49:36.556071 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.555982    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c2454e-79694dx6xb"]
Oct 30 15:49:36.721920 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.721882    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" path="/var/lib/kubelet/pods/34b1f336-1bba-4443-8651-7fbf30a4107c/volumes"
Oct 30 15:49:36.722313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:49:36.722298    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" path="/var/lib/kubelet/pods/cfcb8d4f-6b25-4461-a621-3e39dc831515/volumes"
Oct 30 15:49:59.629274 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:49:59.629191    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:50:29.631197 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:50:29.631158    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:50:48.094847 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.094800    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"]
Oct 30 15:50:48.095307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095274    5064 cpu_manager.go:414] "containerMap: removing container" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.095307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095288    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.095307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095297    5064 cpu_manager.go:414] "containerMap: removing container" podUID="6b268107-de4f-4217-8463-2d075821e072" containerName="main"
Oct 30 15:50:48.095307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095302    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b268107-de4f-4217-8463-2d075821e072" containerName="main"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095317    5064 cpu_manager.go:414] "containerMap: removing container" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095323    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095342    5064 cpu_manager.go:414] "containerMap: removing container" podUID="7f0e258e-1085-413e-afce-21e4e28eca64" containerName="maas-api"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095348    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0e258e-1085-413e-afce-21e4e28eca64" containerName="maas-api"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095355    5064 cpu_manager.go:414] "containerMap: removing container" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095359    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095373    5064 cpu_manager.go:414] "containerMap: removing container" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing"
Oct 30 15:50:48.095446 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095378    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095453    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0e258e-1085-413e-afce-21e4e28eca64" containerName="maas-api"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095462    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcb8d4f-6b25-4461-a621-3e39dc831515" containerName="payload-pre-processing"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095471    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b1f336-1bba-4443-8651-7fbf30a4107c" containerName="payload-processing"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095479    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095484    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b268107-de4f-4217-8463-2d075821e072" containerName="main"
Oct 30 15:50:48.095694 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.095491    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cc6c2d-d2b4-49fe-a8a2-3d4f315a1f53" containerName="istio-proxy"
Oct 30 15:50:48.098733 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.098712    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.107956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.107932    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"]
Oct 30 15:50:48.119412 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.119379    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wvm9s\""
Oct 30 15:50:48.191127 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.191087    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qlh\" (UniqueName: \"kubernetes.io/projected/5cc85b82-4c1d-479e-9844-51a767c21053-kube-api-access-69qlh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.191326 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.191156    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc85b82-4c1d-479e-9844-51a767c21053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.292393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.292347    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qlh\" (UniqueName: \"kubernetes.io/projected/5cc85b82-4c1d-479e-9844-51a767c21053-kube-api-access-69qlh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.292578 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.292417    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc85b82-4c1d-479e-9844-51a767c21053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.292789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.292767    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cc85b82-4c1d-479e-9844-51a767c21053-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.317706 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.317664    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qlh\" (UniqueName: \"kubernetes.io/projected/5cc85b82-4c1d-479e-9844-51a767c21053-kube-api-access-69qlh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-jnzsv\" (UID: \"5cc85b82-4c1d-479e-9844-51a767c21053\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.412282 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.412246    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.739858 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.739821    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"]
Oct 30 15:50:48.742498 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:50:48.742466    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc85b82_4c1d_479e_9844_51a767c21053.slice/crio-db9c67043487fae73655b8e14dc20b7c6ff24784f442b74bd7dbd0e5f88a5c1b WatchSource:0}: Error finding container db9c67043487fae73655b8e14dc20b7c6ff24784f442b74bd7dbd0e5f88a5c1b: Status 404 returned error can't find the container with id db9c67043487fae73655b8e14dc20b7c6ff24784f442b74bd7dbd0e5f88a5c1b
Oct 30 15:50:48.845743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.845698    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv" event={"ID":"5cc85b82-4c1d-479e-9844-51a767c21053","Type":"ContainerStarted","Data":"b6c9fbffaf1f1cdbfdedaf3a3a9f5b806ddb2efdc71ee24f12bef38c0ddec795"}
Oct 30 15:50:48.845956 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.845767    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv" event={"ID":"5cc85b82-4c1d-479e-9844-51a767c21053","Type":"ContainerStarted","Data":"db9c67043487fae73655b8e14dc20b7c6ff24784f442b74bd7dbd0e5f88a5c1b"}
Oct 30 15:50:48.846026 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.845954    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:50:48.893971 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:48.893914    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv" podStartSLOduration=0.893899273 podStartE2EDuration="893.899273ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:48.891696135 +0000 UTC m=+3798.333586853" watchObservedRunningTime="2026-10-30 15:50:48.893899273 +0000 UTC m=+3798.335789989"
Oct 30 15:50:59.629949 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:50:59.629911    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:50:59.853362 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:50:59.853332    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-jnzsv"
Oct 30 15:51:26.863588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:26.863222    5064 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2sncc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:51:26.863588 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:26.863292    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2sncc" podUID="5234d7a7-d39b-431b-a017-aa0498cb5b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:51:29.630657 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:51:29.630622    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:51:40.273393 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.273350    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:51:40.278188 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.278155    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.310742 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.310703    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-22f590-openshift-default-dockercfg-vhm96\""
Oct 30 15:51:40.313121 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.313085    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:51:40.345806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.345771    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.345998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.345840    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.345998 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.345870    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.345991    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346077 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.346037    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346148 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.346118    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.346152    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346187 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.346174    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.346258 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.346229    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpgh\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447287 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447248    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447331    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447442    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447496    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447527    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447553    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpgh\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447620    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.447709 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447674    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447717    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447748    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448042 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.447902    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.448184    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448400 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.448376    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.448906 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.448880    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.450715 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.450687    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.450856 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.450810    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.477922 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.477896    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.478066 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.477952    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpgh\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\") pod \"e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") " pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.595893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.595769    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:40.917826 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.917801    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:51:40.920521 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:51:40.920488    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2877c0f0_a2e4_47f6_b116_9da32cdb3324.slice/crio-a7f320142b0752638f123990d0619487e25e2c37ba601f2f7405fb969b8cd66b WatchSource:0}: Error finding container a7f320142b0752638f123990d0619487e25e2c37ba601f2f7405fb969b8cd66b: Status 404 returned error can't find the container with id a7f320142b0752638f123990d0619487e25e2c37ba601f2f7405fb969b8cd66b
Oct 30 15:51:40.922840 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.922796    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:40.922915 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.922880    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:40.922955 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:40.922912    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:41.261224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:41.261136    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerStarted","Data":"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"}
Oct 30 15:51:41.261224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:41.261195    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerStarted","Data":"a7f320142b0752638f123990d0619487e25e2c37ba601f2f7405fb969b8cd66b"}
Oct 30 15:51:41.321410 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:41.321350    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podStartSLOduration=1.321332283 podStartE2EDuration="1.321332283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:41.318588313 +0000 UTC m=+3850.760479030" watchObservedRunningTime="2026-10-30 15:51:41.321332283 +0000 UTC m=+3850.763223002"
Oct 30 15:51:41.596163 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:41.596071    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:41.601690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:41.601658    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:42.266191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:42.266157    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:42.267321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:42.267295    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:51:45.544619 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.544581    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:51:45.549123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.549092    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.567191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.567162    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-22f590-serving-cert\""
Oct 30 15:51:45.568738 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.568706    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:51:45.605427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.605394    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmw49\" (UniqueName: \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.605633 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.605548    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.605722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.605639    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.605722 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.605682    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.706888 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.706824    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.707086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.706989    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.707086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.707033    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.707209 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.707138    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmw49\" (UniqueName: \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.707613 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.707578    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.709567 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.709545    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.709676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.709600    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.732224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.732201    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmw49\" (UniqueName: \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\") pod \"maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.792464 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.792429    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:51:45.796429 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.796377    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:45.815107 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.815073    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5c72f2b5-dockercfg-gkbrb\""
Oct 30 15:51:45.819466 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.816772    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-22f590\""
Oct 30 15:51:45.824582 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.824549    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:51:45.824739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.824662    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:51:45.830355 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.830330    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:45.843332 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.843297    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:51:45.866034 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.866002    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:45.909064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.909018    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wsj\" (UniqueName: \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:45.909257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.909142    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:45.909257 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.909171    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwbz\" (UniqueName: \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:45.909381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:45.909290    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.011310 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.011273    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.011529 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.011507    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wsj\" (UniqueName: \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.011639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.011625    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:46.011704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.011658    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwbz\" (UniqueName: \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:46.012590 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.012563    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.013374 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.013349    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:46.056764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.056687    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwbz\" (UniqueName: \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\") pod \"payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:46.056764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.056687    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wsj\" (UniqueName: \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\") pod \"payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.110981 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.110928    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:46.154909 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.154871    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:46.224369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.224056    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:51:46.235438 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:51:46.235407    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8356e31_d25a_436b_b5f7_6c7464b6cd36.slice/crio-b60142482e93fef6107b6580d5ba8691d92ad8f905d8c2edb65369acee506dc9 WatchSource:0}: Error finding container b60142482e93fef6107b6580d5ba8691d92ad8f905d8c2edb65369acee506dc9: Status 404 returned error can't find the container with id b60142482e93fef6107b6580d5ba8691d92ad8f905d8c2edb65369acee506dc9
Oct 30 15:51:46.289204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.289167    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" event={"ID":"e8356e31-d25a-436b-b5f7-6c7464b6cd36","Type":"ContainerStarted","Data":"b60142482e93fef6107b6580d5ba8691d92ad8f905d8c2edb65369acee506dc9"}
Oct 30 15:51:46.421764 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.418109    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:51:46.421764 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:51:46.420329    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5689bc52_1e96_4bf4_96f6_de9c088ce005.slice/crio-601471acc02cf63ca02a62aad93322cf88cef3b4f74ec7de417f7dde18d655a5 WatchSource:0}: Error finding container 601471acc02cf63ca02a62aad93322cf88cef3b4f74ec7de417f7dde18d655a5: Status 404 returned error can't find the container with id 601471acc02cf63ca02a62aad93322cf88cef3b4f74ec7de417f7dde18d655a5
Oct 30 15:51:46.472750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:46.472719    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:51:46.475093 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:51:46.475064    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf285cf5_117e_40a9_8e10_7a5514dc404b.slice/crio-d8611c694f63bfcb0cdd9113fa8b5cbcbe6771e0c35d9fb7c68b6e4c9857bf81 WatchSource:0}: Error finding container d8611c694f63bfcb0cdd9113fa8b5cbcbe6771e0c35d9fb7c68b6e4c9857bf81: Status 404 returned error can't find the container with id d8611c694f63bfcb0cdd9113fa8b5cbcbe6771e0c35d9fb7c68b6e4c9857bf81
Oct 30 15:51:47.297755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.297704    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" event={"ID":"e8356e31-d25a-436b-b5f7-6c7464b6cd36","Type":"ContainerStarted","Data":"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"}
Oct 30 15:51:47.298283 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.297865    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:47.299502 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.299471    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" event={"ID":"cf285cf5-117e-40a9-8e10-7a5514dc404b","Type":"ContainerStarted","Data":"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"}
Oct 30 15:51:47.299643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.299525    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:47.299643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.299549    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" event={"ID":"cf285cf5-117e-40a9-8e10-7a5514dc404b","Type":"ContainerStarted","Data":"d8611c694f63bfcb0cdd9113fa8b5cbcbe6771e0c35d9fb7c68b6e4c9857bf81"}
Oct 30 15:51:47.301396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.301366    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" event={"ID":"5689bc52-1e96-4bf4-96f6-de9c088ce005","Type":"ContainerStarted","Data":"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"}
Oct 30 15:51:47.301519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.301417    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" event={"ID":"5689bc52-1e96-4bf4-96f6-de9c088ce005","Type":"ContainerStarted","Data":"601471acc02cf63ca02a62aad93322cf88cef3b4f74ec7de417f7dde18d655a5"}
Oct 30 15:51:47.301519 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.301458    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:47.356561 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.356506    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" podStartSLOduration=1.959575284 podStartE2EDuration="2.356492743s" totalImagesPullingTime="396.917459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:47.354710553 +0000 UTC m=+3856.796601272" watchObservedRunningTime="2026-10-30 15:51:47.356492743 +0000 UTC m=+3856.798383459"
Oct 30 15:51:47.404899 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.404810    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" podStartSLOduration=2.4047901400000002 podStartE2EDuration="2.40479014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:47.403067614 +0000 UTC m=+3856.844958331" watchObservedRunningTime="2026-10-30 15:51:47.40479014 +0000 UTC m=+3856.846680858"
Oct 30 15:51:47.450117 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:47.450058    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" podStartSLOduration=2.450038417 podStartE2EDuration="2.450038417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:47.449328262 +0000 UTC m=+3856.891218979" watchObservedRunningTime="2026-10-30 15:51:47.450038417 +0000 UTC m=+3856.891929134"
Oct 30 15:51:53.326081 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:53.326040    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:51:58.309028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:58.308968    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 15:51:58.309569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:58.309043    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 15:51:58.310625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:58.310602    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:51:58.311074 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:58.311049    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:51:59.391351 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.391318    5064 generic.go:356] "Generic (PLEG): container finished" podID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerID="17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3" exitCode=0
Oct 30 15:51:59.391886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.391365    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerDied","Data":"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"}
Oct 30 15:51:59.391886 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.391743    5064 scope.go:122] "RemoveContainer" containerID="17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"
Oct 30 15:51:59.393181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.393147    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:59.393314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.393235    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:59.393314 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:51:59.393285    5064 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 15:51:59.634149 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:51:59.634110    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:52:00.399806 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:00.399767    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerStarted","Data":"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"}
Oct 30 15:52:00.596496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:00.596453    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:00.596711 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:00.596689    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 30 15:52:00.596774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:00.596737    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 30 15:52:01.450179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.450135    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:01.453531 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.453504    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.479126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.479075    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:01.485523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.485491    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-22f590\"/\"kube-root-ca.crt\""
Oct 30 15:52:01.485523 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.485489    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-22f590\"/\"openshift-service-ca.crt\""
Oct 30 15:52:01.553674 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.553635    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-22f590\"/\"simulator-single-main-22f590-kserve-self-signed-certs\""
Oct 30 15:52:01.553921 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.553692    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-22f590\"/\"default-dockercfg-wmzml\""
Oct 30 15:52:01.597224 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.597177    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 30 15:52:01.597424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.597253    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634425    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634532    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634606    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634665    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfcq\" (UniqueName: \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634692    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.636855 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.634748    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738457    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738563    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738633    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738688    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfcq\" (UniqueName: \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738717    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.738910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.738770    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.739381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.739298    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.749853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.739572    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.749853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.740847    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.750857 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.750525    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.752319 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.752293    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.765549 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.765515    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfcq\" (UniqueName: \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\") pod \"simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") " pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:01.783407 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:01.783368    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:02.146181 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.146025    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:02.150573 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:52:02.150539    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36349e5a_ea0e_44d7_9281_b43b35191aed.slice/crio-92cecc6e7a0826d7c4ff34a5520cdd90044b9f51c245ab331d56c47a877cee51 WatchSource:0}: Error finding container 92cecc6e7a0826d7c4ff34a5520cdd90044b9f51c245ab331d56c47a877cee51: Status 404 returned error can't find the container with id 92cecc6e7a0826d7c4ff34a5520cdd90044b9f51c245ab331d56c47a877cee51
Oct 30 15:52:02.411748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.411655    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" event={"ID":"36349e5a-ea0e-44d7-9281-b43b35191aed","Type":"ContainerStarted","Data":"8053973b2a1f239e5d049b91635b972eaffa125723f1006f5ade30aab7b8c24b"}
Oct 30 15:52:02.411748 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.411697    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" event={"ID":"36349e5a-ea0e-44d7-9281-b43b35191aed","Type":"ContainerStarted","Data":"92cecc6e7a0826d7c4ff34a5520cdd90044b9f51c245ab331d56c47a877cee51"}
Oct 30 15:52:02.462881 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.462793    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" podStartSLOduration=1.462778742 podStartE2EDuration="1.462778742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:02.460952493 +0000 UTC m=+3871.902843191" watchObservedRunningTime="2026-10-30 15:52:02.462778742 +0000 UTC m=+3871.904669457"
Oct 30 15:52:02.596397 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.596361    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 30 15:52:02.596585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.596422    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 30 15:52:02.784426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:02.784333    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:03.596904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:03.596869    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 30 15:52:03.597324 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:03.596923    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 30 15:52:04.596137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:04.596095    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 30 15:52:04.596338 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:04.596159    5064 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 30 15:52:06.084045 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:06.083983    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:06.084683 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:06.084656    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:06.085381 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:06.085356    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:11.783895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:11.783857    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:11.797132 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:11.797101    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:12.499060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:12.499027    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:29.636369 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:52:29.636285    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:52:31.471995 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:31.471962    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:52:31.502750 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:31.502718    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:52:34.494123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:34.494086    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:40.169866 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:40.169810    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:52:40.202989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:40.202953    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:52:41.957484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:41.957387    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:52:41.957990 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:41.957681    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" podUID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" containerName="maas-api" containerID="cri-o://c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163" gracePeriod=30
Oct 30 15:52:42.481063 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.481033    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:52:42.599149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.599062    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\") pod \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") "
Oct 30 15:52:42.599329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.599205    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\") pod \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") "
Oct 30 15:52:42.599329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.599236    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\") pod \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") "
Oct 30 15:52:42.599329 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.599259    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\" (UniqueName: \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\") pod \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\" (UID: \"e8356e31-d25a-436b-b5f7-6c7464b6cd36\") "
Oct 30 15:52:42.599675 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.599639    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca" pod "e8356e31-d25a-436b-b5f7-6c7464b6cd36" (UID: "e8356e31-d25a-436b-b5f7-6c7464b6cd36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:52:42.601369 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.601341    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls" pod "e8356e31-d25a-436b-b5f7-6c7464b6cd36" (UID: "e8356e31-d25a-436b-b5f7-6c7464b6cd36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:52:42.601485 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.601401    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs" pod "e8356e31-d25a-436b-b5f7-6c7464b6cd36" (UID: "e8356e31-d25a-436b-b5f7-6c7464b6cd36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:52:42.601574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.601553    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49" pod "e8356e31-d25a-436b-b5f7-6c7464b6cd36" (UID: "e8356e31-d25a-436b-b5f7-6c7464b6cd36"). InnerVolumeSpecName "kube-api-access-fmw49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:52:42.654114 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.654071    5064 generic.go:356] "Generic (PLEG): container finished" podID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" containerID="c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163" exitCode=0
Oct 30 15:52:42.654336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.654119    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"
Oct 30 15:52:42.654336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.654156    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" event={"ID":"e8356e31-d25a-436b-b5f7-6c7464b6cd36","Type":"ContainerDied","Data":"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"}
Oct 30 15:52:42.654336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.654205    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4" event={"ID":"e8356e31-d25a-436b-b5f7-6c7464b6cd36","Type":"ContainerDied","Data":"b60142482e93fef6107b6580d5ba8691d92ad8f905d8c2edb65369acee506dc9"}
Oct 30 15:52:42.654336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.654227    5064 scope.go:122] "RemoveContainer" containerID="c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"
Oct 30 15:52:42.667023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.667000    5064 scope.go:122] "RemoveContainer" containerID="c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"
Oct 30 15:52:42.667385 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:52:42.667365    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163\": container with ID starting with c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163 not found: ID does not exist" containerID="c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"
Oct 30 15:52:42.667444 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.667395    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163"} err="failed to get container status \"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163\": rpc error: code = NotFound desc = could not find container \"c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163\": container with ID starting with c57b16d4030dbe6d91b9bba4471ee1262dc15f2b159ee01315e45178489b0163 not found: ID does not exist"
Oct 30 15:52:42.701589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.701545    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:42.701589 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.701588    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8356e31-d25a-436b-b5f7-6c7464b6cd36-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:42.701875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.701604    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8356e31-d25a-436b-b5f7-6c7464b6cd36-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:42.701875 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.701624    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmw49\" (UniqueName: \"kubernetes.io/projected/e8356e31-d25a-436b-b5f7-6c7464b6cd36-kube-api-access-fmw49\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:42.721626 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.721593    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:52:42.725903 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:42.725870    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22f590-6fb4ff488f-bt7d4"]
Oct 30 15:52:44.722371 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:44.722333    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" path="/var/lib/kubelet/pods/e8356e31-d25a-436b-b5f7-6c7464b6cd36/volumes"
Oct 30 15:52:45.170256 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:45.170216    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing" containerID="cri-o://59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe" gracePeriod=25
Oct 30 15:52:45.203913 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:45.203869    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing" containerID="cri-o://1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c" gracePeriod=25
Oct 30 15:52:48.309585 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:48.309544    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:52:48.310100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:48.309543    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:52:49.515079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.515011    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" podUID="36349e5a-ea0e-44d7-9281-b43b35191aed" containerName="main" containerID="cri-o://8053973b2a1f239e5d049b91635b972eaffa125723f1006f5ade30aab7b8c24b" gracePeriod=45
Oct 30 15:52:49.706469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.706434    5064 generic.go:356] "Generic (PLEG): container finished" podID="36349e5a-ea0e-44d7-9281-b43b35191aed" containerID="8053973b2a1f239e5d049b91635b972eaffa125723f1006f5ade30aab7b8c24b" exitCode=0
Oct 30 15:52:49.706724 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.706503    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" event={"ID":"36349e5a-ea0e-44d7-9281-b43b35191aed","Type":"ContainerDied","Data":"8053973b2a1f239e5d049b91635b972eaffa125723f1006f5ade30aab7b8c24b"}
Oct 30 15:52:49.789356 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.789327    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:49.887705 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887675    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.887901 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887879    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.887961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887920    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\" (UniqueName: \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.887961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887947    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\" (UniqueName: \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.888087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887988    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.888087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.887992    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:49.888087 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888032    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\") pod \"36349e5a-ea0e-44d7-9281-b43b35191aed\" (UID: \"36349e5a-ea0e-44d7-9281-b43b35191aed\") "
Oct 30 15:52:49.888265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888203    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:49.888334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888278    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:49.888460 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888443    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:49.888536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888460    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:49.888536 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.888472    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:49.890482 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.890458    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:49.890621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.890559    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "kube-api-access-rjfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:52:49.890944 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.890921    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs" pod "36349e5a-ea0e-44d7-9281-b43b35191aed" (UID: "36349e5a-ea0e-44d7-9281-b43b35191aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:52:49.981869 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.981817    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:52:49.982086 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.982059    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" containerID="cri-o://a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5" gracePeriod=30
Oct 30 15:52:49.989118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.989087    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36349e5a-ea0e-44d7-9281-b43b35191aed-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:49.989118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.989121    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfcq\" (UniqueName: \"kubernetes.io/projected/36349e5a-ea0e-44d7-9281-b43b35191aed-kube-api-access-rjfcq\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:49.989288 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:49.989133    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36349e5a-ea0e-44d7-9281-b43b35191aed-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:50.712986 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:50.712956    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"
Oct 30 15:52:50.713465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:50.712955    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz" event={"ID":"36349e5a-ea0e-44d7-9281-b43b35191aed","Type":"ContainerDied","Data":"92cecc6e7a0826d7c4ff34a5520cdd90044b9f51c245ab331d56c47a877cee51"}
Oct 30 15:52:50.713465 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:50.713075    5064 scope.go:122] "RemoveContainer" containerID="8053973b2a1f239e5d049b91635b972eaffa125723f1006f5ade30aab7b8c24b"
Oct 30 15:52:50.772985 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:50.772946    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:50.786586 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:50.786548    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-22f590/simulator-single-main-22f590-kserve-859d6fdb8b-9h8pz"]
Oct 30 15:52:51.086765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:51.086677    5064 patch_prober.go:29] interesting pod/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 15:52:51.086765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:51.086733    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 15:52:52.722331 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:52.722289    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36349e5a-ea0e-44d7-9281-b43b35191aed" path="/var/lib/kubelet/pods/36349e5a-ea0e-44d7-9281-b43b35191aed/volumes"
Oct 30 15:52:54.676912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.676876    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:52:54.677370 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677355    5064 cpu_manager.go:414] "containerMap: removing container" podUID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" containerName="maas-api"
Oct 30 15:52:54.677416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677370    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" containerName="maas-api"
Oct 30 15:52:54.677416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677390    5064 cpu_manager.go:414] "containerMap: removing container" podUID="36349e5a-ea0e-44d7-9281-b43b35191aed" containerName="main"
Oct 30 15:52:54.677416 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677396    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36349e5a-ea0e-44d7-9281-b43b35191aed" containerName="main"
Oct 30 15:52:54.677515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677485    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8356e31-d25a-436b-b5f7-6c7464b6cd36" containerName="maas-api"
Oct 30 15:52:54.677515 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.677500    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="36349e5a-ea0e-44d7-9281-b43b35191aed" containerName="main"
Oct 30 15:52:54.682110 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.682087    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.700376 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.700343    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:52:54.711862 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.711815    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-20c5ba44-kserve-self-signed-certs\""
Oct 30 15:52:54.840017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.839854    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.840440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.840410    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.840571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.840474    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.840571 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.840528    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkfx\" (UniqueName: \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.840669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.840579    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.840669 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.840646    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942056 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.941951    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942071    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942123    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942154    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942180    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkfx\" (UniqueName: \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942210    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942475    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942540 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942493    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.942647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.942576    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.944689 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.944658    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.944883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.944865    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:54.967480 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:54.967446    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkfx\" (UniqueName: \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\") pod \"e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") " pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:55.005491 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.005450    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:55.248494 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.248465    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:55.311532 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.311508    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:52:55.314186 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:52:55.314157    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8afaa9a_c027_48ce_b926_5466878445ac.slice/crio-12157f780f4a2e3c2c1ffdc96c9462ae62859d3f3b34342a0f0ebb5ffeda8c55 WatchSource:0}: Error finding container 12157f780f4a2e3c2c1ffdc96c9462ae62859d3f3b34342a0f0ebb5ffeda8c55: Status 404 returned error can't find the container with id 12157f780f4a2e3c2c1ffdc96c9462ae62859d3f3b34342a0f0ebb5ffeda8c55
Oct 30 15:52:55.345682 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345650    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.345878 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345726    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.345941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345878    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.345941 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345911    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345942    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.345978    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346001    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:55.346051 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346030    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346091    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346092    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:55.346214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346115    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\") pod \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\" (UID: \"2877c0f0-a2e4-47f6-b116-9da32cdb3324\") "
Oct 30 15:52:55.346346 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346263    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:55.347064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346499    5064 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-data\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.347064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346522    5064 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-credential-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.347064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346537    5064 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-socket\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.347064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.346992    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:52:55.347064 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.347019    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:55.348807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.348773    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "kube-api-access-8lpgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:52:55.349007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.348985    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:52:55.349241 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.349212    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 15:52:55.349359 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.349280    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy" pod "2877c0f0-a2e4-47f6-b116-9da32cdb3324" (UID: "2877c0f0-a2e4-47f6-b116-9da32cdb3324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:52:55.448175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448133    5064 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-envoy\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.448175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448174    5064 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2877c0f0-a2e4-47f6-b116-9da32cdb3324-workload-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.448396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448189    5064 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-token\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.448396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448202    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpgh\" (UniqueName: \"kubernetes.io/projected/2877c0f0-a2e4-47f6-b116-9da32cdb3324-kube-api-access-8lpgh\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.448396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448215    5064 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istiod-ca-cert\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.448396 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.448230    5064 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2877c0f0-a2e4-47f6-b116-9da32cdb3324-istio-podinfo\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:52:55.743215 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.743173    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" event={"ID":"b8afaa9a-c027-48ce-b926-5466878445ac","Type":"ContainerStarted","Data":"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"}
Oct 30 15:52:55.743741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.743230    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" event={"ID":"b8afaa9a-c027-48ce-b926-5466878445ac","Type":"ContainerStarted","Data":"12157f780f4a2e3c2c1ffdc96c9462ae62859d3f3b34342a0f0ebb5ffeda8c55"}
Oct 30 15:52:55.745028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.744998    5064 generic.go:356] "Generic (PLEG): container finished" podID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerID="a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5" exitCode=0
Oct 30 15:52:55.745191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.745040    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"
Oct 30 15:52:55.745191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.745063    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerDied","Data":"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"}
Oct 30 15:52:55.745191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.745102    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5" event={"ID":"2877c0f0-a2e4-47f6-b116-9da32cdb3324","Type":"ContainerDied","Data":"a7f320142b0752638f123990d0619487e25e2c37ba601f2f7405fb969b8cd66b"}
Oct 30 15:52:55.745191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.745125    5064 scope.go:122] "RemoveContainer" containerID="a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"
Oct 30 15:52:55.755358 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.755332    5064 scope.go:122] "RemoveContainer" containerID="17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"
Oct 30 15:52:55.764517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.764498    5064 scope.go:122] "RemoveContainer" containerID="a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"
Oct 30 15:52:55.764792 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:52:55.764767    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5\": container with ID starting with a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5 not found: ID does not exist" containerID="a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"
Oct 30 15:52:55.764890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.764803    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5"} err="failed to get container status \"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5\": rpc error: code = NotFound desc = could not find container \"a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5\": container with ID starting with a1582f53d4aaf16f8f8487c76837a8747a3b3f7b570c9ad1d0638d23784ed5b5 not found: ID does not exist"
Oct 30 15:52:55.764890 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.764822    5064 scope.go:122] "RemoveContainer" containerID="17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"
Oct 30 15:52:55.765105 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:52:55.765085    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3\": container with ID starting with 17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3 not found: ID does not exist" containerID="17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"
Oct 30 15:52:55.765172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.765112    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3"} err="failed to get container status \"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3\": rpc error: code = NotFound desc = could not find container \"17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3\": container with ID starting with 17ebf6460118a571aa0cbad94547f3d9f3a91ed71c17b31848073468d358e7f3 not found: ID does not exist"
Oct 30 15:52:55.793018 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.792961    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" podStartSLOduration=1.792946013 podStartE2EDuration="1.792946013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:55.790762014 +0000 UTC m=+3925.232652731" watchObservedRunningTime="2026-10-30 15:52:55.792946013 +0000 UTC m=+3925.234836734"
Oct 30 15:52:55.830807 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.830769    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:52:55.835080 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:55.835046    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22f590-openshift-default-585956f74fsbpv5"]
Oct 30 15:52:56.006307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:56.006230    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:52:56.723334 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:56.723300    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" path="/var/lib/kubelet/pods/2877c0f0-a2e4-47f6-b116-9da32cdb3324/volumes"
Oct 30 15:52:58.321852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:58.320939    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:52:58.321852 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:52:58.321431    5064 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 15:52:59.834191 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:52:59.834152    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:53:05.006432 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.006392    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:53:05.019566 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.019540    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:53:05.474932 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.474904    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:53:05.501169 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.501144    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:53:05.615022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.614936    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\" (UniqueName: \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\") pod \"cf285cf5-117e-40a9-8e10-7a5514dc404b\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") "
Oct 30 15:53:05.615022 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.614991    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\") pod \"5689bc52-1e96-4bf4-96f6-de9c088ce005\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") "
Oct 30 15:53:05.615221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.615134    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\") pod \"cf285cf5-117e-40a9-8e10-7a5514dc404b\" (UID: \"cf285cf5-117e-40a9-8e10-7a5514dc404b\") "
Oct 30 15:53:05.615221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.615166    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\" (UniqueName: \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\") pod \"5689bc52-1e96-4bf4-96f6-de9c088ce005\" (UID: \"5689bc52-1e96-4bf4-96f6-de9c088ce005\") "
Oct 30 15:53:05.615476 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.615445    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume" pod "5689bc52-1e96-4bf4-96f6-de9c088ce005" (UID: "5689bc52-1e96-4bf4-96f6-de9c088ce005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:53:05.615544 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.615488    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume" pod "cf285cf5-117e-40a9-8e10-7a5514dc404b" (UID: "cf285cf5-117e-40a9-8e10-7a5514dc404b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:53:05.617264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.617237    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz" pod "5689bc52-1e96-4bf4-96f6-de9c088ce005" (UID: "5689bc52-1e96-4bf4-96f6-de9c088ce005"). InnerVolumeSpecName "kube-api-access-tnwbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:53:05.617333 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.617271    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj" pod "cf285cf5-117e-40a9-8e10-7a5514dc404b" (UID: "cf285cf5-117e-40a9-8e10-7a5514dc404b"). InnerVolumeSpecName "kube-api-access-22wsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:53:05.716440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.716403    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf285cf5-117e-40a9-8e10-7a5514dc404b-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:05.716440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.716435    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwbz\" (UniqueName: \"kubernetes.io/projected/5689bc52-1e96-4bf4-96f6-de9c088ce005-kube-api-access-tnwbz\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:05.716440 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.716445    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wsj\" (UniqueName: \"kubernetes.io/projected/cf285cf5-117e-40a9-8e10-7a5514dc404b-kube-api-access-22wsj\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:05.716690 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.716454    5064 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5689bc52-1e96-4bf4-96f6-de9c088ce005-plugins-config-volume\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:05.818350 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.818317    5064 generic.go:356] "Generic (PLEG): container finished" podID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerID="59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe" exitCode=0
Oct 30 15:53:05.818547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.818366    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"
Oct 30 15:53:05.818547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.818404    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" event={"ID":"cf285cf5-117e-40a9-8e10-7a5514dc404b","Type":"ContainerDied","Data":"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"}
Oct 30 15:53:05.818547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.818469    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk" event={"ID":"cf285cf5-117e-40a9-8e10-7a5514dc404b","Type":"ContainerDied","Data":"d8611c694f63bfcb0cdd9113fa8b5cbcbe6771e0c35d9fb7c68b6e4c9857bf81"}
Oct 30 15:53:05.818547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.818502    5064 scope.go:122] "RemoveContainer" containerID="59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"
Oct 30 15:53:05.819765 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.819747    5064 generic.go:356] "Generic (PLEG): container finished" podID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerID="1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c" exitCode=0
Oct 30 15:53:05.819863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.819787    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"
Oct 30 15:53:05.819863 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.819819    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" event={"ID":"5689bc52-1e96-4bf4-96f6-de9c088ce005","Type":"ContainerDied","Data":"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"}
Oct 30 15:53:05.819962 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.819888    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7" event={"ID":"5689bc52-1e96-4bf4-96f6-de9c088ce005","Type":"ContainerDied","Data":"601471acc02cf63ca02a62aad93322cf88cef3b4f74ec7de417f7dde18d655a5"}
Oct 30 15:53:05.830650 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.830545    5064 scope.go:122] "RemoveContainer" containerID="59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"
Oct 30 15:53:05.834431 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:53:05.831439    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe\": container with ID starting with 59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe not found: ID does not exist" containerID="59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"
Oct 30 15:53:05.834431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.831476    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe"} err="failed to get container status \"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe\": rpc error: code = NotFound desc = could not find container \"59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe\": container with ID starting with 59117e5c5ce6de3051f12d4cb87d5ad115a351a36e4b95c38a13ac9011247bbe not found: ID does not exist"
Oct 30 15:53:05.834431 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.831500    5064 scope.go:122] "RemoveContainer" containerID="1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"
Oct 30 15:53:05.835893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.835871    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:53:05.843794 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.843776    5064 scope.go:122] "RemoveContainer" containerID="1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"
Oct 30 15:53:05.844180 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:53:05.844157    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c\": container with ID starting with 1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c not found: ID does not exist" containerID="1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"
Oct 30 15:53:05.844254 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.844189    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c"} err="failed to get container status \"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c\": rpc error: code = NotFound desc = could not find container \"1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c\": container with ID starting with 1490ddc9d25405fcf5eacd92613566cacb155d2dfddaa468b091b86f092b0d4c not found: ID does not exist"
Oct 30 15:53:05.876137 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.876103    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:53:05.882643 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.882616    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22f590-f8f54695bjjhgk"]
Oct 30 15:53:05.919808 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.919777    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:53:05.924325 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:05.924295    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22f590-57bb8fjrr7"]
Oct 30 15:53:06.721785 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:06.721749    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" path="/var/lib/kubelet/pods/5689bc52-1e96-4bf4-96f6-de9c088ce005/volumes"
Oct 30 15:53:06.722214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:06.722199    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" path="/var/lib/kubelet/pods/cf285cf5-117e-40a9-8e10-7a5514dc404b/volumes"
Oct 30 15:53:11.667417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.667377    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:53:11.668159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668116    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:53:11.668159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668144    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668183    5064 cpu_manager.go:414] "containerMap: removing container" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668192    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668207    5064 cpu_manager.go:414] "containerMap: removing container" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668216    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668373    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668391    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="5689bc52-1e96-4bf4-96f6-de9c088ce005" containerName="payload-pre-processing"
Oct 30 15:53:11.668405 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668403    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf285cf5-117e-40a9-8e10-7a5514dc404b" containerName="payload-processing"
Oct 30 15:53:11.669028 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.668414    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:53:11.671904 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.671882    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.690741 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.690704    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:53:11.701118 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.701077    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-20c5ba44-kserve-self-signed-certs\""
Oct 30 15:53:11.780262 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780223    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.780461 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780277    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svs4n\" (UniqueName: \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.780568 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780546    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.780621 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780599    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.780679 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780631    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.780802 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.780784    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.881996    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882052    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882105    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882264 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882159    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882625 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882599    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882685 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882637    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882743 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882631    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svs4n\" (UniqueName: \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.882801 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.882782    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.883182 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.883159    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.888774 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.888739    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.890229 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.890197    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.909252 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.909220    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svs4n\" (UniqueName: \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\") pod \"e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") " pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:11.988306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:11.988212    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:12.300301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:12.300266    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:53:12.301652 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:53:12.301625    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc777d8fe_eb72_4932_bcb7_00ad2378cefd.slice/crio-c2cd0047ab0ce046f84155e6982ac6ba277656548d7c8353f27fafa89a032439 WatchSource:0}: Error finding container c2cd0047ab0ce046f84155e6982ac6ba277656548d7c8353f27fafa89a032439: Status 404 returned error can't find the container with id c2cd0047ab0ce046f84155e6982ac6ba277656548d7c8353f27fafa89a032439
Oct 30 15:53:12.860421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:12.860374    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" event={"ID":"c777d8fe-eb72-4932-bcb7-00ad2378cefd","Type":"ContainerStarted","Data":"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"}
Oct 30 15:53:12.860421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:12.860424    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" event={"ID":"c777d8fe-eb72-4932-bcb7-00ad2378cefd","Type":"ContainerStarted","Data":"c2cd0047ab0ce046f84155e6982ac6ba277656548d7c8353f27fafa89a032439"}
Oct 30 15:53:12.906265 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:12.906208    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" podStartSLOduration=1.906193483 podStartE2EDuration="1.906193483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:12.905161556 +0000 UTC m=+3942.347052274" watchObservedRunningTime="2026-10-30 15:53:12.906193483 +0000 UTC m=+3942.348084194"
Oct 30 15:53:12.988661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:12.988622    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:21.988999 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:21.988964    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:22.001547 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:22.001517    5064 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:22.987763 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:22.987728    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:53:29.637724 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:53:29.637675    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:53:39.513419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:39.513377    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:53:45.688843 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:45.688792    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:53:54.535004 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.534956    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" podUID="b8afaa9a-c027-48ce-b926-5466878445ac" containerName="main" containerID="cri-o://582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e" gracePeriod=45
Oct 30 15:53:54.810194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.810163    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:53:54.887992 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.887960    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888041    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888063    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888144    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\" (UniqueName: \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888204 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888174    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888427 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888224    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\") pod \"b8afaa9a-c027-48ce-b926-5466878445ac\" (UID: \"b8afaa9a-c027-48ce-b926-5466878445ac\") "
Oct 30 15:53:54.888892 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.888848    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:53:54.889069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.889024    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:53:54.889069 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.889045    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:53:54.891436 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.891396    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:53:54.891574 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.891537    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "kube-api-access-vrkfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:53:54.891691 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.891661    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs" pod "b8afaa9a-c027-48ce-b926-5466878445ac" (UID: "b8afaa9a-c027-48ce-b926-5466878445ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:53:54.989378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989331    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8afaa9a-c027-48ce-b926-5466878445ac-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:54.989378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989364    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:54.989378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989372    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:54.989378 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989383    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkfx\" (UniqueName: \"kubernetes.io/projected/b8afaa9a-c027-48ce-b926-5466878445ac-kube-api-access-vrkfx\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:54.989672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989394    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:54.989672 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:54.989405    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8afaa9a-c027-48ce-b926-5466878445ac-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:53:55.330938 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.330897    5064 generic.go:356] "Generic (PLEG): container finished" podID="b8afaa9a-c027-48ce-b926-5466878445ac" containerID="582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e" exitCode=0
Oct 30 15:53:55.331157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.330950    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"
Oct 30 15:53:55.331157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.330960    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" event={"ID":"b8afaa9a-c027-48ce-b926-5466878445ac","Type":"ContainerDied","Data":"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"}
Oct 30 15:53:55.331157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.331009    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv" event={"ID":"b8afaa9a-c027-48ce-b926-5466878445ac","Type":"ContainerDied","Data":"12157f780f4a2e3c2c1ffdc96c9462ae62859d3f3b34342a0f0ebb5ffeda8c55"}
Oct 30 15:53:55.331157 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.331033    5064 scope.go:122] "RemoveContainer" containerID="582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"
Oct 30 15:53:55.341406 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.341386    5064 scope.go:122] "RemoveContainer" containerID="582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"
Oct 30 15:53:55.341724 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:53:55.341704    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e\": container with ID starting with 582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e not found: ID does not exist" containerID="582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"
Oct 30 15:53:55.341776 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.341737    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e"} err="failed to get container status \"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e\": rpc error: code = NotFound desc = could not find container \"582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e\": container with ID starting with 582672da07d919ab084dcf51810033fa00468b3591908ed8ba0e0258598f8a1e not found: ID does not exist"
Oct 30 15:53:55.392179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.392144    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:53:55.401198 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:55.401171    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-20c5ba44-kserve-85f5475df7-n28pv"]
Oct 30 15:53:56.722190 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:53:56.722156    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8afaa9a-c027-48ce-b926-5466878445ac" path="/var/lib/kubelet/pods/b8afaa9a-c027-48ce-b926-5466878445ac/volumes"
Oct 30 15:53:59.631740 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:53:59.631701    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:54:00.709799 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:00.709758    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" podUID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" containerName="main" containerID="cri-o://6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da" gracePeriod=45
Oct 30 15:54:00.977251 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:00.977186    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:54:01.054111 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054074    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\" (UniqueName: \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054126    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054154    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054208    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054321 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054245    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054546 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054352    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\" (UniqueName: \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\") pod \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\" (UID: \"c777d8fe-eb72-4932-bcb7-00ad2378cefd\") "
Oct 30 15:54:01.054777 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054743    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:54:01.054910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054791    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:54:01.054910 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.054809    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:54:01.056787 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.056759    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 15:54:01.056935 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.056899    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:54:01.057010 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.056990    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n" pod "c777d8fe-eb72-4932-bcb7-00ad2378cefd" (UID: "c777d8fe-eb72-4932-bcb7-00ad2378cefd"). InnerVolumeSpecName "kube-api-access-svs4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:54:01.156496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156454    5064 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tls-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.156496 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156497    5064 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-dshm\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.156731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156510    5064 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-tmp-dir\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.156731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156525    5064 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-home\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.156731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156539    5064 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c777d8fe-eb72-4932-bcb7-00ad2378cefd-model-cache\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.156731 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.156569    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svs4n\" (UniqueName: \"kubernetes.io/projected/c777d8fe-eb72-4932-bcb7-00ad2378cefd-kube-api-access-svs4n\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:01.363070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.362981    5064 generic.go:356] "Generic (PLEG): container finished" podID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" containerID="6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da" exitCode=0
Oct 30 15:54:01.363070 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.363036    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"
Oct 30 15:54:01.363268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.363041    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" event={"ID":"c777d8fe-eb72-4932-bcb7-00ad2378cefd","Type":"ContainerDied","Data":"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"}
Oct 30 15:54:01.363268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.363160    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p" event={"ID":"c777d8fe-eb72-4932-bcb7-00ad2378cefd","Type":"ContainerDied","Data":"c2cd0047ab0ce046f84155e6982ac6ba277656548d7c8353f27fafa89a032439"}
Oct 30 15:54:01.363268 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.363188    5064 scope.go:122] "RemoveContainer" containerID="6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"
Oct 30 15:54:01.373647 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.373627    5064 scope.go:122] "RemoveContainer" containerID="6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"
Oct 30 15:54:01.373926 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:54:01.373905    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da\": container with ID starting with 6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da not found: ID does not exist" containerID="6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"
Oct 30 15:54:01.373980 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.373936    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da"} err="failed to get container status \"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da\": rpc error: code = NotFound desc = could not find container \"6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da\": container with ID starting with 6aac91725ac761775d57a74ca41e170f4552b9df44f62b20146b63382ecdb8da not found: ID does not exist"
Oct 30 15:54:01.426122 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.426088    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:54:01.439730 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:01.439681    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-20c5ba44-kserve-74d8545d65-7lv2p"]
Oct 30 15:54:02.721612 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:02.721573    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" path="/var/lib/kubelet/pods/c777d8fe-eb72-4932-bcb7-00ad2378cefd/volumes"
Oct 30 15:54:22.442477 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442440    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-t9622"]
Oct 30 15:54:22.442961 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442946    5064 cpu_manager.go:414] "containerMap: removing container" podUID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" containerName="main"
Oct 30 15:54:22.443007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442962    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" containerName="main"
Oct 30 15:54:22.443007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442975    5064 cpu_manager.go:414] "containerMap: removing container" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:54:22.443007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442980    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2877c0f0-a2e4-47f6-b116-9da32cdb3324" containerName="istio-proxy"
Oct 30 15:54:22.443007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.442997    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b8afaa9a-c027-48ce-b926-5466878445ac" containerName="main"
Oct 30 15:54:22.443007 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.443006    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8afaa9a-c027-48ce-b926-5466878445ac" containerName="main"
Oct 30 15:54:22.443173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.443089    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="c777d8fe-eb72-4932-bcb7-00ad2378cefd" containerName="main"
Oct 30 15:54:22.443173 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.443102    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8afaa9a-c027-48ce-b926-5466878445ac" containerName="main"
Oct 30 15:54:22.446462 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.446440    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.468035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.467914    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-t9622"]
Oct 30 15:54:22.512654 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.512623    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc4m\" (UniqueName: \"kubernetes.io/projected/c1b59c8d-4092-4305-9425-93ce31b663f7-kube-api-access-gjc4m\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.512845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.512701    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b59c8d-4092-4305-9425-93ce31b663f7-openshift-service-ca\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.512845 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.512732    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-maas-api-tls\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.512937 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.512862    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-metrics-certs\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.614168 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.614122    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-metrics-certs\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.614379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.614238    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc4m\" (UniqueName: \"kubernetes.io/projected/c1b59c8d-4092-4305-9425-93ce31b663f7-kube-api-access-gjc4m\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.614379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.614319    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b59c8d-4092-4305-9425-93ce31b663f7-openshift-service-ca\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.614379 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.614347    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-maas-api-tls\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.615174 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.615145    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b59c8d-4092-4305-9425-93ce31b663f7-openshift-service-ca\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.616893 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.616867    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-metrics-certs\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.617082 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.617011    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b59c8d-4092-4305-9425-93ce31b663f7-maas-api-tls\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.639556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.639525    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc4m\" (UniqueName: \"kubernetes.io/projected/c1b59c8d-4092-4305-9425-93ce31b663f7-kube-api-access-gjc4m\") pod \"maas-api-595cc648c-t9622\" (UID: \"c1b59c8d-4092-4305-9425-93ce31b663f7\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:22.763558 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:22.763471    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:23.072577 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:23.072542    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-t9622"]
Oct 30 15:54:23.075122 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:54:23.075089    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b59c8d_4092_4305_9425_93ce31b663f7.slice/crio-6a41ff7f89163b600015ec9ae2816811a8ff2449d3ee097d0f21b23034ddffb7 WatchSource:0}: Error finding container 6a41ff7f89163b600015ec9ae2816811a8ff2449d3ee097d0f21b23034ddffb7: Status 404 returned error can't find the container with id 6a41ff7f89163b600015ec9ae2816811a8ff2449d3ee097d0f21b23034ddffb7
Oct 30 15:54:23.076988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:23.076956    5064 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:54:23.496347 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:23.496313    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622" event={"ID":"c1b59c8d-4092-4305-9425-93ce31b663f7","Type":"ContainerStarted","Data":"6a41ff7f89163b600015ec9ae2816811a8ff2449d3ee097d0f21b23034ddffb7"}
Oct 30 15:54:24.503755 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:24.503709    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622" event={"ID":"c1b59c8d-4092-4305-9425-93ce31b663f7","Type":"ContainerStarted","Data":"6f8cdf027c0231f8f28761482935ad46e1390c42b608ad40bab30f9c2775f20a"}
Oct 30 15:54:24.504307 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:24.503857    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:24.563100 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:24.563037    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622" podStartSLOduration=2.092813296 podStartE2EDuration="2.563023754s" totalImagesPullingTime="470.210458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:24.561439455 +0000 UTC m=+4014.003330198" watchObservedRunningTime="2026-10-30 15:54:24.563023754 +0000 UTC m=+4014.004914471"
Oct 30 15:54:29.633703 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:54:29.633663    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:54:30.514817 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:30.514785    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-t9622"
Oct 30 15:54:30.630994 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:30.630957    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 15:54:30.631221 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:30.631171    5064 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" podUID="b1336714-66b2-4de3-9dc9-03828e5da7ac" containerName="maas-api" containerID="cri-o://1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08" gracePeriod=30
Oct 30 15:54:32.019584 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.019260    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 15:54:32.123488 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.123438    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\") pod \"b1336714-66b2-4de3-9dc9-03828e5da7ac\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") "
Oct 30 15:54:32.123671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.123513    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\") pod \"b1336714-66b2-4de3-9dc9-03828e5da7ac\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") "
Oct 30 15:54:32.123671 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.123618    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\" (UniqueName: \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\") pod \"b1336714-66b2-4de3-9dc9-03828e5da7ac\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") "
Oct 30 15:54:32.123752 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.123675    5064 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\") pod \"b1336714-66b2-4de3-9dc9-03828e5da7ac\" (UID: \"b1336714-66b2-4de3-9dc9-03828e5da7ac\") "
Oct 30 15:54:32.124443 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.124405    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca" pod "b1336714-66b2-4de3-9dc9-03828e5da7ac" (UID: "b1336714-66b2-4de3-9dc9-03828e5da7ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:54:32.126474 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.126433    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs" pod "b1336714-66b2-4de3-9dc9-03828e5da7ac" (UID: "b1336714-66b2-4de3-9dc9-03828e5da7ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:54:32.126685 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.126656    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls" pod "b1336714-66b2-4de3-9dc9-03828e5da7ac" (UID: "b1336714-66b2-4de3-9dc9-03828e5da7ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:54:32.127134 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.127108    5064 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b" pod "b1336714-66b2-4de3-9dc9-03828e5da7ac" (UID: "b1336714-66b2-4de3-9dc9-03828e5da7ac"). InnerVolumeSpecName "kube-api-access-qtm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:54:32.225160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.225120    5064 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-metrics-certs\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:32.225160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.225156    5064 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1336714-66b2-4de3-9dc9-03828e5da7ac-maas-api-tls\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:32.225160 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.225179    5064 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtm7b\" (UniqueName: \"kubernetes.io/projected/b1336714-66b2-4de3-9dc9-03828e5da7ac-kube-api-access-qtm7b\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:32.225522 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.225194    5064 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1336714-66b2-4de3-9dc9-03828e5da7ac-openshift-service-ca\") on node \"ip-10-0-138-143.ec2.internal\" DevicePath \"\""
Oct 30 15:54:32.551076 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.551028    5064 generic.go:356] "Generic (PLEG): container finished" podID="b1336714-66b2-4de3-9dc9-03828e5da7ac" containerID="1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08" exitCode=0
Oct 30 15:54:32.551306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.551094    5064 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"
Oct 30 15:54:32.551306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.551229    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" event={"ID":"b1336714-66b2-4de3-9dc9-03828e5da7ac","Type":"ContainerDied","Data":"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"}
Oct 30 15:54:32.551306 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.551278    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r" event={"ID":"b1336714-66b2-4de3-9dc9-03828e5da7ac","Type":"ContainerDied","Data":"6f6a22963a2c38bacd45c1c49c9731f11e1cda6365544b2e4122bbb4c8864f75"}
Oct 30 15:54:32.551517 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.551311    5064 scope.go:122] "RemoveContainer" containerID="1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"
Oct 30 15:54:32.569123 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.569096    5064 scope.go:122] "RemoveContainer" containerID="1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"
Oct 30 15:54:32.569561 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:54:32.569530    5064 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08\": container with ID starting with 1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08 not found: ID does not exist" containerID="1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"
Oct 30 15:54:32.569677 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.569580    5064 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08"} err="failed to get container status \"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08\": rpc error: code = NotFound desc = could not find container \"1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08\": container with ID starting with 1ff1fcca02ecf3ab0755ebd89d1309d617de3b272f96a9d3845dcd49e40bcc08 not found: ID does not exist"
Oct 30 15:54:32.610451 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.610409    5064 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 15:54:32.634495 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.634444    5064 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hlm6r"]
Oct 30 15:54:32.731853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:54:32.731565    5064 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1336714-66b2-4de3-9dc9-03828e5da7ac" path="/var/lib/kubelet/pods/b1336714-66b2-4de3-9dc9-03828e5da7ac/volumes"
Oct 30 15:54:59.630051 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:54:59.630006    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:55:29.631409 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:55:29.631362    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:55:31.929498 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:31.929461    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-65cd8c9c54-sqsrv_cdc8a5aa-fcda-4035-88cb-de3efce8b081/manager/0.log"
Oct 30 15:55:32.827815 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:32.827778    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:55:34.271159 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:34.271128    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-jknvj_3897e788-0c71-4f61-8820-11015ce1ac74/copy-manifests/0.log"
Oct 30 15:55:34.313060 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:34.313028    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-jknvj_3897e788-0c71-4f61-8820-11015ce1ac74/manager/0.log"
Oct 30 15:55:34.551676 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:34.551595    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-5dxn5_7505b204-f9df-471b-94b8-546ae5d5c907/proxy/0.log"
Oct 30 15:55:35.545965 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:35.545929    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-t9622_c1b59c8d-4092-4305-9425-93ce31b663f7/maas-api/0.log"
Oct 30 15:55:37.435299 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:37.435264    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-46kbm_91d59170-b98e-471e-8dbb-ac0dfc15c071/postgres/0.log"
Oct 30 15:55:39.213240 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:39.213208    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-fnqxl_599621e7-347f-4568-bead-91a824451b43/manager/0.log"
Oct 30 15:55:39.860175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:39.860138    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-jnzsv_5cc85b82-4c1d-479e-9844-51a767c21053/manager/0.log"
Oct 30 15:55:41.080556 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:41.080457    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2qv8f_d47b1eb7-c394-4aa6-8969-75a8269a0d02/istio-proxy/0.log"
Oct 30 15:55:41.696179 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:41.696140    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-wk65g_afabb0f7-990c-4aea-8d89-cfa11b587cff/kube-auth-proxy/0.log"
Oct 30 15:55:41.894418 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:41.894386    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-2sncc_5234d7a7-d39b-431b-a017-aa0498cb5b33/istio-proxy/0.log"
Oct 30 15:55:42.306116 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:42.306082    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9vjxl_982473c9-6215-4df5-8a3f-6d88ffbac6e4/istio-proxy/1.log"
Oct 30 15:55:42.520702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:42.520669    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-bfgkh_d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a/istio-proxy/0.log"
Oct 30 15:55:43.111149 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:43.111116    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-pg2k9_59fca057-7aa9-41ad-aaec-f00a0a6754a0/istio-proxy/0.log"
Oct 30 15:55:43.926136 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:43.926094    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fwvxh_2c109ff3-12cb-4096-a383-20e111d550f8/payload-pre-processing/0.log"
Oct 30 15:55:44.119421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:44.119382    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:55:44.321217 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:44.321137    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58ddf469d5-lvlsc_9e34f75c-8c90-48eb-b5e6-4d363017a5f3/router/0.log"
Oct 30 15:55:44.779569 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:44.779533    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9vjxl_982473c9-6215-4df5-8a3f-6d88ffbac6e4/istio-proxy/0.log"
Oct 30 15:55:45.309417 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:45.309384    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hptln_d20bc29c-6f82-4373-87c2-dd32b724e7dd/main/0.log"
Oct 30 15:55:45.705524 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:45.705494    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9qz5r_f009e6e3-5135-48c0-864c-c9672de9b17a/main/0.log"
Oct 30 15:55:46.117670 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:46.117641    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx_034782e3-3b6e-4c3b-852b-ea4b23f06a75/main/0.log"
Oct 30 15:55:56.791154 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:56.791109    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-65cd8c9c54-sqsrv_cdc8a5aa-fcda-4035-88cb-de3efce8b081/manager/0.log"
Oct 30 15:55:57.743450 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:57.743413    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:55:59.229822 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:59.229794    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-jknvj_3897e788-0c71-4f61-8820-11015ce1ac74/copy-manifests/0.log"
Oct 30 15:55:59.277884 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:59.277851    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-jknvj_3897e788-0c71-4f61-8820-11015ce1ac74/manager/0.log"
Oct 30 15:55:59.513336 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:55:59.513249    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-5dxn5_7505b204-f9df-471b-94b8-546ae5d5c907/proxy/0.log"
Oct 30 15:55:59.631094 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:55:59.631053    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:56:00.282242 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:00.282205    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-t9622_c1b59c8d-4092-4305-9425-93ce31b663f7/maas-api/0.log"
Oct 30 15:56:02.148355 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:02.148329    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-46kbm_91d59170-b98e-471e-8dbb-ac0dfc15c071/postgres/0.log"
Oct 30 15:56:03.604341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:03.604307    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-fnqxl_599621e7-347f-4568-bead-91a824451b43/manager/0.log"
Oct 30 15:56:04.237040 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:04.237008    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-jnzsv_5cc85b82-4c1d-479e-9844-51a767c21053/manager/0.log"
Oct 30 15:56:05.274017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:05.273974    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2qv8f_d47b1eb7-c394-4aa6-8969-75a8269a0d02/istio-proxy/0.log"
Oct 30 15:56:05.851977 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:05.851932    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-wk65g_afabb0f7-990c-4aea-8d89-cfa11b587cff/kube-auth-proxy/0.log"
Oct 30 15:56:06.046267 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:06.046241    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-2sncc_5234d7a7-d39b-431b-a017-aa0498cb5b33/istio-proxy/0.log"
Oct 30 15:56:06.457639 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:06.457606    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9vjxl_982473c9-6215-4df5-8a3f-6d88ffbac6e4/istio-proxy/1.log"
Oct 30 15:56:06.662504 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:06.662468    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-bfgkh_d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a/istio-proxy/0.log"
Oct 30 15:56:07.233425 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:07.233386    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-pg2k9_59fca057-7aa9-41ad-aaec-f00a0a6754a0/istio-proxy/0.log"
Oct 30 15:56:08.052604 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:08.052546    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fwvxh_2c109ff3-12cb-4096-a383-20e111d550f8/payload-pre-processing/0.log"
Oct 30 15:56:08.243696 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:08.243661    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:56:08.454269 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:08.454231    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58ddf469d5-lvlsc_9e34f75c-8c90-48eb-b5e6-4d363017a5f3/router/0.log"
Oct 30 15:56:09.017788 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:09.017757    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hptln_d20bc29c-6f82-4373-87c2-dd32b724e7dd/main/0.log"
Oct 30 15:56:09.442180 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:09.442141    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9qz5r_f009e6e3-5135-48c0-864c-c9672de9b17a/main/0.log"
Oct 30 15:56:09.854341 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:09.854253    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxvtx_034782e3-3b6e-4c3b-852b-ea4b23f06a75/main/0.log"
Oct 30 15:56:29.630253 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:56:29.630212    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:56:44.406606 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:44.406568    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:56:44.765175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:44.765073    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:56:45.454580 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:45.454549    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:56:45.652434 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:56:45.652406    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:56:59.628487 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:56:59.628450    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:57:29.630304 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:57:29.630260    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:57:29.849861 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:29.849805    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:57:30.193272 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:30.193239    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68588d554c-9tr9l_88625478-51b5-4938-a65c-8665f086e8b2/manager/0.log"
Oct 30 15:57:31.003514 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.003486    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:57:31.205172 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.205145    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:57:31.507422 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.507391    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:57:31.536736 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.536710    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:57:31.563510 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.563474    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cbkvm/must-gather-z862k"]
Oct 30 15:57:31.563966 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.563951    5064 cpu_manager.go:414] "containerMap: removing container" podUID="b1336714-66b2-4de3-9dc9-03828e5da7ac" containerName="maas-api"
Oct 30 15:57:31.564017 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.563967    5064 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1336714-66b2-4de3-9dc9-03828e5da7ac" containerName="maas-api"
Oct 30 15:57:31.564079 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.564070    5064 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1336714-66b2-4de3-9dc9-03828e5da7ac" containerName="maas-api"
Oct 30 15:57:31.567273 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.567253    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.584132 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.584093    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"kube-root-ca.crt\""
Oct 30 15:57:31.597895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.597864    5064 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"openshift-service-ca.crt\""
Oct 30 15:57:31.597895 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.597889    5064 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cbkvm\"/\"default-dockercfg-wlh7f\""
Oct 30 15:57:31.614883 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.614847    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/must-gather-z862k"]
Oct 30 15:57:31.710126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.710080    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq9l\" (UniqueName: \"kubernetes.io/projected/babd4e2f-e93f-4680-b126-230ba2b44bbd-kube-api-access-svq9l\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.710126 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.710129    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/babd4e2f-e93f-4680-b126-230ba2b44bbd-must-gather-output\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.811814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.811713    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq9l\" (UniqueName: \"kubernetes.io/projected/babd4e2f-e93f-4680-b126-230ba2b44bbd-kube-api-access-svq9l\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.811814 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.811770    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/babd4e2f-e93f-4680-b126-230ba2b44bbd-must-gather-output\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.812194 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.812172    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/babd4e2f-e93f-4680-b126-230ba2b44bbd-must-gather-output\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.841469 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.841435    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq9l\" (UniqueName: \"kubernetes.io/projected/babd4e2f-e93f-4680-b126-230ba2b44bbd-kube-api-access-svq9l\") pod \"must-gather-z862k\" (UID: \"babd4e2f-e93f-4680-b126-230ba2b44bbd\") " pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:31.895704 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:31.895666    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/must-gather-z862k"
Oct 30 15:57:32.202335 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:32.202310    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/must-gather-z862k"]
Oct 30 15:57:32.204723 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:57:32.204697    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbabd4e2f_e93f_4680_b126_230ba2b44bbd.slice/crio-8a7cdbcf208cf5bb12a5747b20416006e0ec4615b23f46a7cf28826121fbba48 WatchSource:0}: Error finding container 8a7cdbcf208cf5bb12a5747b20416006e0ec4615b23f46a7cf28826121fbba48: Status 404 returned error can't find the container with id 8a7cdbcf208cf5bb12a5747b20416006e0ec4615b23f46a7cf28826121fbba48
Oct 30 15:57:32.594413 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:32.594327    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/must-gather-z862k" event={"ID":"babd4e2f-e93f-4680-b126-230ba2b44bbd","Type":"ContainerStarted","Data":"8a7cdbcf208cf5bb12a5747b20416006e0ec4615b23f46a7cf28826121fbba48"}
Oct 30 15:57:33.604424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:33.604327    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/must-gather-z862k" event={"ID":"babd4e2f-e93f-4680-b126-230ba2b44bbd","Type":"ContainerStarted","Data":"204ab5c345249803e16f8f2299abbc639adfd62783be7899c8c29eccc16c0a95"}
Oct 30 15:57:33.604424 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:33.604385    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/must-gather-z862k" event={"ID":"babd4e2f-e93f-4680-b126-230ba2b44bbd","Type":"ContainerStarted","Data":"ed12249af93b43bedd16c8f77a0354fa5cb3412f68511c2b32fa2437d283e64b"}
Oct 30 15:57:33.649092 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:33.649032    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cbkvm/must-gather-z862k" podStartSLOduration=1.9199747390000002 podStartE2EDuration="2.649017292s" totalImagesPullingTime="729.042553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:33.646515927 +0000 UTC m=+4203.088406637" watchObservedRunningTime="2026-10-30 15:57:33.649017292 +0000 UTC m=+4203.090908050"
Oct 30 15:57:36.599185 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:36.599140    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pnhqm_9a72ab8d-d509-46b8-809f-55a708439ffd/global-pull-secret-syncer/0.log"
Oct 30 15:57:36.790912 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:36.790881    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kfr99_26527e26-b484-446f-b704-ee332fea2f2e/connection-checker/0.log"
Oct 30 15:57:36.926849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:36.926546    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lvqzv_ebdd075b-3f66-4f77-ab73-e8f080e78f37/connection-checker/0.log"
Oct 30 15:57:37.068849 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:37.065586    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tlhw9_ed3aca72-3ba7-4f4d-9fd1-114a98c3daf8/connection-checker/0.log"
Oct 30 15:57:37.411853 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:37.411712    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k4c5w_c6c8e73f-c556-4dfe-b3b8-8b3e60dc98d3/konnectivity-agent/0.log"
Oct 30 15:57:37.697313 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:37.693028    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-143.ec2.internal_d6501707c434eeb14bc7e15cb32ecb7b/haproxy/0.log"
Oct 30 15:57:43.964646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:43.964612    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-fnqxl_599621e7-347f-4568-bead-91a824451b43/manager/0.log"
Oct 30 15:57:44.411348 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:44.411309    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-jnzsv_5cc85b82-4c1d-479e-9844-51a767c21053/manager/0.log"
Oct 30 15:57:46.228426 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.227487    5064 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"]
Oct 30 15:57:46.234230 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.234198    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.253988 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.253955    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"]
Oct 30 15:57:46.311131 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.311086    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-podres\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.311337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.311140    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-sys\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.311337 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.311232    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-proc\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.311439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.311360    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-lib-modules\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.311439 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.311417    5064 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpb9\" (UniqueName: \"kubernetes.io/projected/4b1eb183-cae0-4d78-af91-7d6882267076-kube-api-access-4tpb9\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412223    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-podres\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412271 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412265    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-sys\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412291    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-proc\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412324    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-lib-modules\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412365    5064 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpb9\" (UniqueName: \"kubernetes.io/projected/4b1eb183-cae0-4d78-af91-7d6882267076-kube-api-access-4tpb9\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412366    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-podres\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412412    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-sys\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412412    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-proc\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.412539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.412468    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b1eb183-cae0-4d78-af91-7d6882267076-lib-modules\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.437433 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.437396    5064 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpb9\" (UniqueName: \"kubernetes.io/projected/4b1eb183-cae0-4d78-af91-7d6882267076-kube-api-access-4tpb9\") pod \"perf-node-gather-daemonset-djhrs\" (UID: \"4b1eb183-cae0-4d78-af91-7d6882267076\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.559421 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.559339    5064 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:46.869214 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:46.869184    5064 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"]
Oct 30 15:57:46.872128 ip-10-0-138-143 kubenswrapper[5064]: W1030 15:57:46.872087    5064 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b1eb183_cae0_4d78_af91_7d6882267076.slice/crio-6c162d5e81067a10cd2a7d00702328aabc474fbabd9c1dbd989e7e84c798c42e WatchSource:0}: Error finding container 6c162d5e81067a10cd2a7d00702328aabc474fbabd9c1dbd989e7e84c798c42e: Status 404 returned error can't find the container with id 6c162d5e81067a10cd2a7d00702328aabc474fbabd9c1dbd989e7e84c798c42e
Oct 30 15:57:47.106539 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:47.106499    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-dtg77_d375dcba-a3e9-40ec-9e41-fb7e692430f4/cluster-monitoring-operator/0.log"
Oct 30 15:57:47.743175 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:47.743138    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs" event={"ID":"4b1eb183-cae0-4d78-af91-7d6882267076","Type":"ContainerStarted","Data":"75a62695e36020e61abddf7268b18e17206876038c54726925f630a91d6a97d7"}
Oct 30 15:57:47.743646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:47.743193    5064 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs" event={"ID":"4b1eb183-cae0-4d78-af91-7d6882267076","Type":"ContainerStarted","Data":"6c162d5e81067a10cd2a7d00702328aabc474fbabd9c1dbd989e7e84c798c42e"}
Oct 30 15:57:47.743646 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:47.743351    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:47.789398 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:47.789337    5064 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs" podStartSLOduration=1.789322431 podStartE2EDuration="1.789322431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:47.787299988 +0000 UTC m=+4217.229190705" watchObservedRunningTime="2026-10-30 15:57:47.789322431 +0000 UTC m=+4217.231213147"
Oct 30 15:57:48.618867 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:48.616884    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgxtr_97b28f19-4139-4144-98af-3a0a302b2e16/node-exporter/0.log"
Oct 30 15:57:48.746035 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:48.746000    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgxtr_97b28f19-4139-4144-98af-3a0a302b2e16/kube-rbac-proxy/0.log"
Oct 30 15:57:48.880165 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:48.880089    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgxtr_97b28f19-4139-4144-98af-3a0a302b2e16/init-textfile/0.log"
Oct 30 15:57:50.554943 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:50.554906    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-v6pj9_7f83a575-78ea-4cc2-8bec-82f21f20beb9/prometheus-operator-admission-webhook/0.log"
Oct 30 15:57:53.222652 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:53.222619    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-m9l7n_bfad0d99-650b-4984-8afb-757f92bf7db9/networking-console-plugin/0.log"
Oct 30 15:57:53.760250 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:53.760217    5064 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-djhrs"
Oct 30 15:57:54.384939 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:54.384902    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/1.log"
Oct 30 15:57:54.392235 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:54.392195    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7j8cn_f3374e02-805e-4d08-97c9-2207b67e62f8/console-operator/2.log"
Oct 30 15:57:55.649997 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:55.649959    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-mgcvn_bc04fbc7-880f-411b-b2bd-0e59c0368d54/download-server/0.log"
Oct 30 15:57:56.942191 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:56.942153    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-qdfhs_30c328c3-1bca-4576-9fd0-c669059e423e/volume-data-source-validator/0.log"
Oct 30 15:57:59.307622 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:59.307584    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fr4t_bef2ba8e-37ad-41c2-a322-b481ba3be1e1/dns/0.log"
Oct 30 15:57:59.434789 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:59.434758    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2qv8f_d47b1eb7-c394-4aa6-8969-75a8269a0d02/istio-proxy/0.log"
Oct 30 15:57:59.444484 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:57:59.444454    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fr4t_bef2ba8e-37ad-41c2-a322-b481ba3be1e1/kube-rbac-proxy/0.log"
Oct 30 15:57:59.632464 ip-10-0-138-143 kubenswrapper[5064]: E1030 15:57:59.632423    5064 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s4n86" containerName="extractor"
Oct 30 15:58:00.343483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:00.343456    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vnzrb_43d22383-d225-4613-baf3-30a0b288519e/dns-node-resolver/0.log"
Oct 30 15:58:01.527023 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.526989    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dd64b66d-fn2zw_05be3936-5126-4d01-ba32-06cb852138da/registry/0.log"
Oct 30 15:58:01.611483 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.611450    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-wk65g_afabb0f7-990c-4aea-8d89-cfa11b587cff/kube-auth-proxy/0.log"
Oct 30 15:58:01.680739 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.680706    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96lz7_9cce1e62-9cf4-44b6-8f63-9dd948c5f0b4/node-ca/0.log"
Oct 30 15:58:01.682419 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.682394    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-2sncc_5234d7a7-d39b-431b-a017-aa0498cb5b33/istio-proxy/0.log"
Oct 30 15:58:01.822011 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.821933    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9vjxl_982473c9-6215-4df5-8a3f-6d88ffbac6e4/istio-proxy/0.log"
Oct 30 15:58:01.824301 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.824280    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9vjxl_982473c9-6215-4df5-8a3f-6d88ffbac6e4/istio-proxy/1.log"
Oct 30 15:58:01.864916 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:01.864869    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-bfgkh_d8e5ffd6-da5b-4fc6-a400-cfa4acf4887a/istio-proxy/0.log"
Oct 30 15:58:02.073717 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:02.073614    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-pg2k9_59fca057-7aa9-41ad-aaec-f00a0a6754a0/istio-proxy/0.log"
Oct 30 15:58:02.360317 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:02.360234    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-fwvxh_2c109ff3-12cb-4096-a383-20e111d550f8/payload-pre-processing/0.log"
Oct 30 15:58:02.428661 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:02.428626    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5z4d5_78b17b36-bfcb-4f6e-aba5-995d1012fd99/payload-processing/0.log"
Oct 30 15:58:02.514702 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:02.514665    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58ddf469d5-lvlsc_9e34f75c-8c90-48eb-b5e6-4d363017a5f3/router/0.log"
Oct 30 15:58:03.852989 ip-10-0-138-143 kubenswrapper[5064]: I1030 15:58:03.852959    5064 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2qv8f_d47b1eb7-c394-4aa6-8969-75a8269a0d02/istio-proxy/0.log"